March 2021 - openstack-discuss - lists.openstack.org

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

5 7

Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
by Christian Rohmann 28 Jun '22

28 Jun '22

Hello Openstack-Discuss, after some digging I found out that cinder learned to resize attached (in-use) volumes quite a while ago with the introduction of the initial "extend 'in-use' volume" feature: https://review.opendev.org/c/openstack/cinder/+/454287/ The support was then extended to also cover Ceph RBD backed volumes with: https://review.opendev.org/c/openstack/nova/+/613039/ Since this is only about the cinder part, I was wondering if nova would ever find out and would actively rescan the device / volume and quintessentially present an increased new size to the guest. Apparently this is where a certain volume-extended event comes into play: https://review.opendev.org/c/openstack/nova/+/454322/ that is to be emitted by cinder when a volume had been extened. I then went head and tried this with the openstack cli (as Horizon does not seem to support / offer to resize in-use volumes. I run Openstack Train with Ceph RBD as storage. First I ran into an issue with the openstack cli (https://bugs.launchpad.net/cinder/+bug/1871759/comments/2 <https://bugs.launchpad.net/cinder/+bug/1871759/comments/2>) but using cinder extend $volumeid $newsize I was able to resize in-use volumes just fine. The only thing missing was the propagation to the guest. I played around with SCSI rescans, but that did not work, reboots also failed to do anyting in this regards. Stopping and starting the VM did work, but why having the capability to online-resize an attached volume when not being able to resize the guest volume as well? Apparently there seems to be an old and somewhat similar observation / bug with https://bugs.launchpad.net/nova/+bug/1369465 So I was simply wondering if this is expected to be working? Are there any special settings / options I need to set to enable this feature? Thanks and with kind regards Christian

4 9

[nova][dev] Revisiting qemu emulation where guest arch != host arch
by Apsey, Christopher 05 Jan '22

05 Jan '22

All, A few years ago I asked a question[1] about why nova, when given a hw_architecture property from glance for an image, would not end up using the correct qemu-system-xx binary when starting the guest process on a compute node if that compute nodes architecture did not match the proposed guest architecture. As an example, if we had all x86 hosts, but wanted to run an emulated ppc guest, we should be able to do that given that at least one compute node had qemu-system-ppc already installed and libvirt was successfully reporting that as a supported architecture to nova. It seemed like a heavy lift at the time, so it was put on the back burner. I am now in a position to fund a contract developer to make this happen, so the question is: would this be a useful blueprint that would potentially be accepted? Most of the time when people want to run an emulated guest they would just nest it inside of an already running guest of the native architecture, but that severely limits observability and the task of managing any more than a handful of instances in this manner quickly becomes a tangled nightmare of networking, etc. I see real benefit in allowing this scenario to run natively so all of the tooling that exists for fleet management 'just works'. This would also be a significant differentiator for OpenStack as a whole. Thoughts? [1] http://lists.openstack.org/pipermail/openstack-operators/2018-August/015653… Chris Apsey Director | Georgia Cyber Range GEORGIA CYBER CENTER 100 Grace Hopper Lane | Augusta, Georgia | 30901 https://www.gacybercenter.org

2 2

Help with eventlet 0.26.1 and dnspython >= 2
by Thomas Goirand 04 Oct '21

04 Oct '21

Hi, As you may know, eventlet is incompatible with dnspython >= 2.0.0.0rc1. See [1] for the details. However, Debian unstable has 2.0.0. Would there be some good soul willing to help me fix this situation? I would need a patch to fix this, but I'm really not sure how to start. Cheers, Thomas Goirand (zigo)

8 15

[all]Introduction to venus which is the project of log management and has been contributed to the OpenStack community
by Liye Pang(逄立业) 10 Sep '21

10 Sep '21

Hello everyone, after feedback from a large number of operations and maintenance personnel in InCloud OpenStack, we developed the log management project “Venus” for the OpenStack projects and that has contributed to the OpenStack community. The following is an introduction to “Venus”. If there is interest in the community, we are interested in proposing it to become an official OpenStack project in the future. Background In the day-to-day operation and maintenance of large-scale cloud platform, the following problems are encountered： l Time-consuming for log querying while the server increasing to thousands. l Difficult to retrieve logs, since there are many modules in the platform, e.g. systems service, compute, storage, network and other platform services. l The large amount and dispersion of log make faults are difficult to be discovered. l Because of distributed and interaction between components of the cloud platform, and scattered logs between components, it will take more time to locate problems. About Venus According to the key requirements of OpenStack in log storage, retrieval, analysis and so on, we introduced Venus project, a unified log management module. This module can provide a one-stop solution to log collection, cleaning, indexing, analysis, alarm, visualization, report generation and other needs, which involves helping operator or maintainer to quickly solve retrieve problems, grasp the operational health of the platform, and improve the management capabilities of the cloud platform. Additionally, this module plans to use machine learning algorithms to quickly locate IT failures and root causes, and improve operation and maintenance efficiency. Application scenario Venus played a key role in the following scenarios： l Retrieval: Provide a simple and easy-to-use way to retrieve all log and the context. l Analysis: Realize log association, field value statistics, and provide multi-scene and multi-dimensional visual analysis reports. l Alerts：Convert retrieval into active alerts to realize the error finding in massive logs. l Issue location: Establish a chain relationship and knowledge graphs to quickly locate problems. Overall structure The architecture of log management system based on Venus and elastic search is as follows: Diagram 0: Architecture of Venus venus_api: API module，provide API、rest-api service. venus_manager: Internal timing task module to realize the core functions of the log system. Current progress The current progress of the Venus project is as follows: l Collection：Develop fluentd collection tasks based on collectd to read, filter, format and send plug-ins for OpenStack, operating systems, and platform services, etc. l Index：Dealing with multi-dimensional index data in elasticsearch, and provide more concise and comprehensive authentication interface to return query results. l Analysis：Analyzing and display the related module errors, Mariadb connection errors, and Rabbitmq connection errors. l Alerts：Develop alarm task code to set threshold for the number of error logs of different modules at different times, and provides alarm services and notification services. l Location：Develop the call chain analysis function based on global_requested series, which can show the execution sequence, time and error information, etc., and provide the export operation. l Management：Develop configuration management functions in the log system, such as alarm threshold setting, timing task management, and log saving time setting, etc. Application examples Two examples of Venus application scenarios are as follows. 1. The virtual machine creation operation was performed on the cloud platform and it was found that the virtual machine was not created successfully. First, we can find the request id of the operation and jump to the virtual machine creation call chain page. Then, we can query the calling process, view and download the details of the log of the call. 2. In the cloud platform, the error log of each module can be converted into alarms to remind the users. Further, we can retrieve the details of the error log and error log statistics. Next step The next step of the Venus project is as follows: * Collection：In addition to fluent, other collection plugins such as logstash will be integrated. * Analysis: Explore more operation and maintenance scenarios, and conduct statistical analysis and alarm on key data. * display: The configuration, analysis and alarm of Venus will be integrated into horizon in the form of plugin. * location: Form clustering log and construct knowledge map, and integrate algorithm class library to locate the root cause of the fault. Venus Project Registry Venus library: https://opendev.org/inspur/venus You can grab the source code using the following git command: git clone https://opendev.org/inspur/venus.git Venus Demo Youtu.be: <https://youtu.be/mE2MoEx3awM> https://youtu.be/mE2MoEx3awM

7 15

Question regarding the production status of DB2 in Trove
by Bekir Fajkovic 31 May '21

31 May '21

Hello! One question regarding the development status of DB2 inside Trove. I see that DB2 is still in experimental phase, when could it be expected for that datastore type to get production ready status? Best Regards. Bekir Fajkovic Senior DBA Mobile: +46 70 019 48 47 www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

3 8

[simplification] Making ask.openstack.org read-only
by Thierry Carrez 27 May '21

27 May '21

Hi everyone, This has been discussed several times on this mailing list in the past, but we never got to actually pull the plug. Ask.openstack.org was launched in 2013. The reason for hosting our own setup was to be able to support multiple languages, while StackOverflow rejected our proposal to have our own openstack-branded StackExchange site. The Chinese ask.o.o side never really took off. The English side also never really worked perfectly (like email alerts are hopelessly broken), but we figured it would get better with time if a big community formed around it. Fast-forward to 2020 and the instance is lacking volunteers to help run it, while the code (and our customization of it) has become more complicated to maintain. It regularly fails one way or another, and questions there often go unanswered, making us look bad. Of the top 30 users, most have abandoned the platform since 2017, leaving only Bernd Bausch actively engaging and helping moderate questions lately. We have called for volunteers several times, but the offers for help never really materialized. At the same time, people are asking OpenStack questions on StackOverflow, and sometimes getting answers there[1]. The fragmentation of the "questions" space is not helping users getting good answers. I think it's time to pull the plug, make ask.openstack.org read-only (so that links to old answers are not lost) and redirect users to the mailing-list and the "OpenStack" tag on StackOverflow. I picked StackOverflow since it seems to have the most openstack questions (2,574 on SO, 76 on SuperUser and 430 on ServerFault). We discussed that option several times, but I now proposed a change to actually make it happen: https://review.opendev.org/#/c/746497/ It's always a difficult decision to make to kill a resource, but I feel like in this case, consolidation and simplification would help. Thoughts, comments? [1] https://stackoverflow.com/questions/tagged/openstack -- Thierry

10 18

[stein][neutron] gratuitous arp
by Ignazio Cassano 14 May '21

14 May '21

Hello, I have a problem on stein neutron. When a vm migrate from one node to another I cannot ping it for several minutes. If in the vm I put a script that ping the gateway continously, the live migration works fine and I can ping it. Why this happens ? I read something about gratuitous arp. Please, help me ? Any workaround , please ? Best Regards Ignazio

3 33

[octavia] victoria - loadbalancer works but its operational status is offline
by apps＠mossakowski.ch 13 May '21

13 May '21

Hello, I have stable/victoria baremetal openstack with octavia installed on centos8 using openvswitch mechanism driver: octavia api on controller, health-manager,housekeeping,worker on 3 compute/network nodes. Official docs include only ubuntu with linuxbridge mechanism but I used https://github.com/prastamaha/openstack-octavia as a reference to get it working on centos8 with ovs. I will push those docs instructions for centos8 soon: https://github.com/openstack/octavia/tree/master/doc/source/install. I created basic http scenario using https://docs.openstack.org/octavia/victoria/user/guides/basic-cookbook.html…. Loadbalancer works but its operational status is offline (openstack_loadbalancer_outputs.txt). On all octavia workers I see the same warning message in health_manager.log: Health Manager experienced an exception processing a heartbeat message from ('172.31.255.233', 1907). Ignoring this packet. Exception: 'NoneType' object has no attribute 'encode' I've searched for related active bug but all I found is this not related in my opinion: https://storyboard.openstack.org/#!/story/2008615 I'm attaching all info I've gathered: - octavia.conf and health_manager debug logs (octavia_config_and_health_manager_logs.txt) - tcpdump from amphora VM (tcpdump_from_amphora_vm.txt) - tcpdump from octavia worker (tcpdump_from_octavia_worker.txt) - debug amphora-agent.log from amphora VM (amphora-agent.log) Can you point me to the right direction what I have missed? Thanks! Piotr Mossakowski https://github.com/moss2k13

4 5

[ironic] Announcing deprecation of the iSCSI deploy interface
by Dmitry Tantsur 04 May '21

04 May '21

Hi all, Following up to the previous mailing list [1] and virtual meetup [2] discussions, I would like to announce the plans to deprecate the 'iscsi' deploy interface. This is the updated plan discussed on the virtual meetup: 1) In the Victoria cycle (i.e. right now): - Fill in the detected feature gaps [3]. - Switch off the iscsi deploy interface by default. - Change [agent]image_dowload_source to HTTP by default. - Give the direct deploy a higher priority, so that it's used by default unless disabled. - Mark it as deprecated in the code (causing warnings when enabled). - Release a major version of ironic to highlight the defaults changes. 2) In the W cycle: - Keep the iscsi deploy deprecated. - Listen to operators' feedback. 3) In the X cycle - Remove the iscsi deploy completely from ironic and IPA. - Remove support code from ironic-lib with a major version bump. Please let us know if you have any questions or concerns. Dmitry [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-August/016681.h… [2] https://etherpad.opendev.org/p/Ironic-Victoria-midcycle [3] https://storyboard.openstack.org/#!/story/2008075 -- Red Hat GmbH, https://de.redhat.com/ , Registered seat: Grasbrunn, Commercial register: Amtsgericht Muenchen, HRB 153243, Managing Directors: Charles Cachera, Brian Klemm, Laurie Krebs, Michael O'Neill

1 1

[release] Meeting Time Poll
by Herve Beraud 03 May '21

03 May '21

Hello We have a few regular attendees of the Release Management meeting who have conflicts with the current meeting time. As a result, we would like to find a new time to hold the meeting. I've created a Doodle poll[1] for everyone to give their input on times. It's mostly limited to times that reasonably overlap the working day in the US and Europe since that's where most of our attendees are located. If you attend the Release Management meeting, please fill out the poll so we can hopefully find a time that works better for everyone. For the sake of organization and to allow everyone to schedule his agenda accordingly, the poll will be closed on April 5th. On that date, I will announce the time of this meeting and the date on which it will take effect. Thanks! [1] https://doodle.com/poll/ip6tg4fvznz7p3qx -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 5

[barbican][oslo][nova][glance][cinder] cursive library status
by Brian Rosmaita 03 May '21

03 May '21

Hello Barbican team, Apologies for not including barbican in the previous thread on this topic: http://lists.openstack.org/pipermail/openstack-discuss/2020-December/019430… The situation is that cursive is used by Nova, Glance, and Cinder and we'd like to move it out of the 'x' namespace into openstack governance. The question is then what team would oversee it. It seems like a good fit for Oslo, and the Oslo team seems OK with that, but since barbican-core is currently included in cursive-core, it make sense to give the Barbican team first dibs. From the consuming teams' side, I don't think we have a preference as long as it's clear who we need to bother about approvals if a bugfix is posted for review. Thus my ask is that the Barbican team indicate whether they'd like to move cursive to the 'openstack' namespace under their governance, or whether they'd prefer Oslo to oversee the library. Thank you! brian

5 5

[Vitrage] Cannot delete old entities on graph
by Adam Tomas 27 Apr '21

27 Apr '21

Hi, I’ve deployed Kolla-Ansible (Victoria) and after using it for some time I have errors from Vitrage service: 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init [-] Got Exception for event {'vitrage_entity_type': 'consistency', 'vitrage_datasource_action': 'update', 'vitrage_sample_date': '2021-03-18T13:30:19Z', 'vitrage_event_type': 'delete_entity', 'vitrage_id': 'a19330f1-49c1-4a97-9787-c80b645f51a6', 'id': '93c1df77-dbb5-4ba7-9b7d-16fad97169f1', 'vitrage_type': 'cinder.volume', 'vitrage_category': 'RESOURCE', 'is_real_vitrage_id': True}: KeyError: 'vitrage_is_placeholder' 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init Traceback (most recent call last): 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/graph_init.py", line 146, in do_work 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init return do_work_func(event) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/graph_init.py", line 111, in process_event 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init self.processor.process_event(event) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/processor/processor.py", line 63, in process_event 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init self.actions[entity.action](entity.vertex, entity.neighbors) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/processor/processor.py", line 149, in delete_entity 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init PUtils.delete_placeholder_vertex(self.entity_graph, vertex) 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/entity_graph/processor/processor_utils.py", line 68, in delete_placeholder_vertex 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init if not vertex[VProps.VITRAGE_IS_PLACEHOLDER]: 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init File "/var/lib/kolla/venv/lib/python3.8/site-packages/vitrage/graph/driver/elements.py", line 24, in __getitem__ 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init return self.properties[key] 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init KeyError: 'vitrage_is_placeholder' 2021-03-18 13:30:19.873 7 ERROR vitrage.entity_graph.graph_init As I can see in Horizon, there are some old volumes , which in Cinder database have proper (DELETED/DETACHED) status, but they’re still visible on Entity Graph, and as in error message above - cannot be deleted by vitrage-graph. How to delete such entities? Best regards, Adam Tomas

2 7

Create OpenStack VMs in few seconds
by open infra 23 Apr '21

23 Apr '21

Hi, I was looking for a way to provision and connect to VMs within a few seconds and ended up in the OpenStack Summit demo on 'OpenStack Virtual Machine Quickly Available in One Second'. I highly appreciate if someone can provide a tutorial or related documentation on how to achieve this? Regards, Danishka

6 16

[kuryr] vPTG April 2021
by Maysa De Macedo Souza 23 Apr '21

23 Apr '21

Hello, Two sessions were scheduled for Kuryr on the upcoming PTG: - 7-8 UTC on April 20 - 2-3 UTC on April 22 Everyone is more than welcome to join the sessions and check our future plans, give feedback or discuss anything regarding Kuryr. Even though participation is free registration is needed[1]. Regards, Maysa Macedo. [1] https://april2021-ptg.eventbrite.com

2 2

[nova] Nominate sean-k-mooney for nova-specs-core
by Stephen Finucane 19 Apr '21

19 Apr '21

Hey, Sean has been working on nova for what seems like yonks now. Each cycle, they spend a significant amount of time reviewing proposed specs and contributing to discussions at the PTG. This is important work and their contributions provide everyone with a deep pool of knowledge on all things networking and hardware upon which to draw. I think the nova project would benefit from their addition to the specs core reviewer team and I therefore propose we add Sean to nova- specs-core. Assuming there are no objections, I'll work with gibi to add Sean to nova-specs- core next week. Cheers, Stephen

9 9

[Murano] Please help with murano image creation
by Bujack, Stefan 19 Apr '21

19 Apr '21

Hello, we want to create the murano image for an openstack victoria deployment. We tried to follow this documentation: https://docs.openstack.org/murano/victoria/reference/appendix/articles/imag… But we were not able to successfully create an image. At first we got the following error: murano-agent requires Python '>=3.6' but the running Python is 2.7.17 Then I changed the release to focal and got another error: Unable to locate package python-pip Is anybody so kind to help me with this problem? Many thanks in advance Greets Stefan Bujack root@kolla-2:~# cat /etc/issue Ubuntu 20.04.2 LTS \n \l root@kolla-2:~# export GITDIR=~/git root@kolla-2:~# mkdir -p $GITDIR root@kolla-2:~# cd $GITDIR root@kolla-2:~/git# git clone https://opendev.org/openstack/murano Cloning into 'murano'... remote: Enumerating objects: 34335, done. remote: Counting objects: 100% (34335/34335), done. remote: Compressing objects: 100% (14856/14856), done. remote: Total 34335 (delta 22813), reused 29217 (delta 18268) Receiving objects: 100% (34335/34335), 16.73 MiB | 7.42 MiB/s, done. Resolving deltas: 100% (22813/22813), done. root@kolla-2:~/git# git clone https://opendev.org/openstack/murano-agent Cloning into 'murano-agent'... remote: Enumerating objects: 6696, done. remote: Counting objects: 100% (6696/6696), done. remote: Compressing objects: 100% (2792/2792), done. remote: Total 6696 (delta 3766), reused 6475 (delta 3586) Receiving objects: 100% (6696/6696), 4.98 MiB | 5.46 MiB/s, done. Resolving deltas: 100% (3766/3766), done. root@kolla-2:~/git# pip3 install diskimage-builder Collecting diskimage-builder Downloading diskimage_builder-3.7.0-py3-none-any.whl (608 kB) |████████████████████████████████| 608 kB 4.8 MB/s Requirement already satisfied: stevedore>=1.20.0 in /usr/lib/python3/dist-packages (from diskimage-builder) (1.32.0) Collecting flake8<4.0.0,>=3.6.0 Downloading flake8-3.8.4-py2.py3-none-any.whl (72 kB) |████████████████████████████████| 72 kB 1.6 MB/s Requirement already satisfied: pbr!=2.1.0,>=2.0.0 in /usr/lib/python3/dist-packages (from diskimage-builder) (5.4.5) Collecting networkx>=1.10 Downloading networkx-2.5-py3-none-any.whl (1.6 MB) |████████████████████████████████| 1.6 MB 7.6 MB/s Requirement already satisfied: PyYAML>=3.12 in /usr/lib/python3/dist-packages (from diskimage-builder) (5.3.1) Collecting pycodestyle<2.7.0,>=2.6.0a1 Downloading pycodestyle-2.6.0-py2.py3-none-any.whl (41 kB) |████████████████████████████████| 41 kB 331 kB/s Collecting pyflakes<2.3.0,>=2.2.0 Downloading pyflakes-2.2.0-py2.py3-none-any.whl (66 kB) |████████████████████████████████| 66 kB 5.8 MB/s Collecting mccabe<0.7.0,>=0.6.0 Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB) Requirement already satisfied: decorator>=4.3.0 in /usr/lib/python3/dist-packages (from networkx>=1.10->diskimage-builder) (4.4.2) Installing collected packages: pycodestyle, pyflakes, mccabe, flake8, networkx, diskimage-builder Successfully installed diskimage-builder-3.7.0 flake8-3.8.4 mccabe-0.6.1 networkx-2.5 pycodestyle-2.6.0 pyflakes-2.2.0 root@kolla-2:~/git# apt install qemu-utils curl tox Reading package lists... Done Building dependency tree Reading state information... Done curl is already the newest version (7.68.0-1ubuntu2.4). curl set to manually installed. The following additional packages will be installed: ibverbs-providers javascript-common libboost-iostreams1.71.0 libboost-thread1.71.0 libibverbs1 libiscsi7 libjs-jquery libjs-sphinxdoc libjs-underscore libnl-route-3-200 librados2 librbd1 librdmacm1 python3-distlib python3-filelock python3-pluggy python3-py python3-toml python3-virtualenv qemu-block-extra sharutils virtualenv Suggested packages: apache2 | lighttpd | httpd subversion python3-pytest debootstrap sharutils-doc bsd-mailx | mailx The following NEW packages will be installed: ibverbs-providers javascript-common libboost-iostreams1.71.0 libboost-thread1.71.0 libibverbs1 libiscsi7 libjs-jquery libjs-sphinxdoc libjs-underscore libnl-route-3-200 librados2 librbd1 librdmacm1 python3-distlib python3-filelock python3-pluggy python3-py python3-toml python3-virtualenv qemu-block-extra qemu-utils sharutils tox virtualenv 0 upgraded, 24 newly installed, 0 to remove and 14 not upgraded. Need to get 8108 kB of archives. After this operation, 37.8 MB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libnl-route-3-200 amd64 3.4.0-1 [149 kB] Get:2 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libibverbs1 amd64 28.0-1ubuntu1 [53.6 kB] Get:3 http://de.archive.ubuntu.com/ubuntu focal/main amd64 ibverbs-providers amd64 28.0-1ubuntu1 [232 kB] Get:4 http://de.archive.ubuntu.com/ubuntu focal/main amd64 javascript-common all 11 [6066 B] Get:5 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libboost-iostreams1.71.0 amd64 1.71.0-6ubuntu6 [237 kB] Get:6 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libboost-thread1.71.0 amd64 1.71.0-6ubuntu6 [249 kB] Get:7 http://de.archive.ubuntu.com/ubuntu focal/main amd64 librdmacm1 amd64 28.0-1ubuntu1 [64.9 kB] Get:8 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libiscsi7 amd64 1.18.0-2 [63.9 kB] Get:9 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libjs-jquery all 3.3.1~dfsg-3 [329 kB] Get:10 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libjs-underscore all 1.9.1~dfsg-1 [98.6 kB] Get:11 http://de.archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] Get:12 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 librados2 amd64 15.2.7-0ubuntu0.20.04.2 [3204 kB] Get:13 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 librbd1 amd64 15.2.7-0ubuntu0.20.04.2 [1609 kB] Get:14 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-distlib all 0.3.0-1 [116 kB] Get:15 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-filelock all 3.0.12-2 [7948 B] Get:16 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] Get:17 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-py all 1.8.1-1 [65.4 kB] Get:18 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-toml all 0.10.0-3 [14.6 kB] Get:19 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 python3-virtualenv all 20.0.17-1 [63.4 kB] Get:20 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-block-extra amd64 1:4.2-3ubuntu6.12 [53.8 kB] Get:21 http://de.archive.ubuntu.com/ubuntu focal-updates/main amd64 qemu-utils amd64 1:4.2-3ubuntu6.12 [974 kB] Get:22 http://de.archive.ubuntu.com/ubuntu focal/main amd64 sharutils amd64 1:4.15.2-4build1 [155 kB] Get:23 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 virtualenv all 20.0.17-1 [2132 B] Get:24 http://de.archive.ubuntu.com/ubuntu focal/universe amd64 tox all 3.13.2-2 [244 kB] Fetched 8108 kB in 0s (31.9 MB/s) Selecting previously unselected package libnl-route-3-200:amd64. (Reading database ... 118797 files and directories currently installed.) Preparing to unpack .../00-libnl-route-3-200_3.4.0-1_amd64.deb ... Unpacking libnl-route-3-200:amd64 (3.4.0-1) ... Selecting previously unselected package libibverbs1:amd64. Preparing to unpack .../01-libibverbs1_28.0-1ubuntu1_amd64.deb ... Unpacking libibverbs1:amd64 (28.0-1ubuntu1) ... Selecting previously unselected package ibverbs-providers:amd64. Preparing to unpack .../02-ibverbs-providers_28.0-1ubuntu1_amd64.deb ... Unpacking ibverbs-providers:amd64 (28.0-1ubuntu1) ... Selecting previously unselected package javascript-common. Preparing to unpack .../03-javascript-common_11_all.deb ... Unpacking javascript-common (11) ... Selecting previously unselected package libboost-iostreams1.71.0:amd64. Preparing to unpack .../04-libboost-iostreams1.71.0_1.71.0-6ubuntu6_amd64.deb ... Unpacking libboost-iostreams1.71.0:amd64 (1.71.0-6ubuntu6) ... Selecting previously unselected package libboost-thread1.71.0:amd64. Preparing to unpack .../05-libboost-thread1.71.0_1.71.0-6ubuntu6_amd64.deb ... Unpacking libboost-thread1.71.0:amd64 (1.71.0-6ubuntu6) ... Selecting previously unselected package librdmacm1:amd64. Preparing to unpack .../06-librdmacm1_28.0-1ubuntu1_amd64.deb ... Unpacking librdmacm1:amd64 (28.0-1ubuntu1) ... Selecting previously unselected package libiscsi7:amd64. Preparing to unpack .../07-libiscsi7_1.18.0-2_amd64.deb ... Unpacking libiscsi7:amd64 (1.18.0-2) ... Selecting previously unselected package libjs-jquery. Preparing to unpack .../08-libjs-jquery_3.3.1~dfsg-3_all.deb ... Unpacking libjs-jquery (3.3.1~dfsg-3) ... Selecting previously unselected package libjs-underscore. Preparing to unpack .../09-libjs-underscore_1.9.1~dfsg-1_all.deb ... Unpacking libjs-underscore (1.9.1~dfsg-1) ... Selecting previously unselected package libjs-sphinxdoc. Preparing to unpack .../10-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... Selecting previously unselected package librados2. Preparing to unpack .../11-librados2_15.2.7-0ubuntu0.20.04.2_amd64.deb ... Unpacking librados2 (15.2.7-0ubuntu0.20.04.2) ... Selecting previously unselected package librbd1. Preparing to unpack .../12-librbd1_15.2.7-0ubuntu0.20.04.2_amd64.deb ... Unpacking librbd1 (15.2.7-0ubuntu0.20.04.2) ... Selecting previously unselected package python3-distlib. Preparing to unpack .../13-python3-distlib_0.3.0-1_all.deb ... Unpacking python3-distlib (0.3.0-1) ... Selecting previously unselected package python3-filelock. Preparing to unpack .../14-python3-filelock_3.0.12-2_all.deb ... Unpacking python3-filelock (3.0.12-2) ... Selecting previously unselected package python3-pluggy. Preparing to unpack .../15-python3-pluggy_0.13.0-2_all.deb ... Unpacking python3-pluggy (0.13.0-2) ... Selecting previously unselected package python3-py. Preparing to unpack .../16-python3-py_1.8.1-1_all.deb ... Unpacking python3-py (1.8.1-1) ... Selecting previously unselected package python3-toml. Preparing to unpack .../17-python3-toml_0.10.0-3_all.deb ... Unpacking python3-toml (0.10.0-3) ... Selecting previously unselected package python3-virtualenv. Preparing to unpack .../18-python3-virtualenv_20.0.17-1_all.deb ... Unpacking python3-virtualenv (20.0.17-1) ... Selecting previously unselected package qemu-block-extra:amd64. Preparing to unpack .../19-qemu-block-extra_1%3a4.2-3ubuntu6.12_amd64.deb ... Unpacking qemu-block-extra:amd64 (1:4.2-3ubuntu6.12) ... Selecting previously unselected package qemu-utils. Preparing to unpack .../20-qemu-utils_1%3a4.2-3ubuntu6.12_amd64.deb ... Unpacking qemu-utils (1:4.2-3ubuntu6.12) ... Selecting previously unselected package sharutils. Preparing to unpack .../21-sharutils_1%3a4.15.2-4build1_amd64.deb ... Unpacking sharutils (1:4.15.2-4build1) ... Selecting previously unselected package virtualenv. Preparing to unpack .../22-virtualenv_20.0.17-1_all.deb ... Unpacking virtualenv (20.0.17-1) ... Selecting previously unselected package tox. Preparing to unpack .../23-tox_3.13.2-2_all.deb ... Unpacking tox (3.13.2-2) ... Setting up javascript-common (11) ... Setting up python3-filelock (3.0.12-2) ... Setting up python3-py (1.8.1-1) ... Setting up python3-distlib (0.3.0-1) ... Setting up libboost-iostreams1.71.0:amd64 (1.71.0-6ubuntu6) ... Setting up libnl-route-3-200:amd64 (3.4.0-1) ... Setting up python3-toml (0.10.0-3) ... Setting up python3-pluggy (0.13.0-2) ... Setting up libboost-thread1.71.0:amd64 (1.71.0-6ubuntu6) ... Setting up libjs-jquery (3.3.1~dfsg-3) ... Setting up sharutils (1:4.15.2-4build1) ... Setting up libjs-underscore (1.9.1~dfsg-1) ... Setting up libibverbs1:amd64 (28.0-1ubuntu1) ... Setting up ibverbs-providers:amd64 (28.0-1ubuntu1) ... Setting up python3-virtualenv (20.0.17-1) ... Setting up virtualenv (20.0.17-1) ... Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... Setting up librdmacm1:amd64 (28.0-1ubuntu1) ... Setting up librados2 (15.2.7-0ubuntu0.20.04.2) ... Setting up tox (3.13.2-2) ... Setting up librbd1 (15.2.7-0ubuntu0.20.04.2) ... Setting up libiscsi7:amd64 (1.18.0-2) ... Setting up qemu-block-extra:amd64 (1:4.2-3ubuntu6.12) ... Setting up qemu-utils (1:4.2-3ubuntu6.12) ... Processing triggers for libc-bin (2.31-0ubuntu9.2) ... Processing triggers for man-db (2.9.1-1) ... Processing triggers for install-info (6.7.0.dfsg.2-5) ... root@kolla-2:~/git# export ELEMENTS_PATH=$GITDIR/murano/contrib/elements:$GITDIR/murano-agent/contrib/elements root@kolla-2:~/git# disk-image-create vm ubuntu murano-agent -o murano-agent.qcow2 2021-02-18 12:52:34.387 | murano-agent requires Python '>=3.6' but the running Python is 2.7.17 root@kolla-2:~/git# export DIB_RELEASE=focal root@kolla-2:~/git# disk-image-create vm ubuntu murano-agent -o murano-agent.qcow2 2021-02-18 12:55:41.222 | E: Unable to locate package python-pip

2 1

[kolla-ansible][horizon] policy.yaml/json files
by Adam Tomas 16 Apr '21

16 Apr '21

Hi, Im not quite clear about policy.yaml/json files in kolla-ansible. Let assume, that I need to allow one of project users to add other users to the project. So I create „project_admin” role and assign it to this user. Then I found /etc/kolla/keystone/policy.json.test file, which I use as template. There is rule „identity:create_credential” : „(role:admin and system_scope:all)” so I add „or role:project_admin” and put file in /etc/kolla/config/keystone/ and reconfigure kolla. And now few questions: 1. policy.json (or policy.yaml) always overwrite all default policies? I mean if I only add one rule to this file then other rules will „disappear” or will have default values? Is there any way to only overwrite some default rules and leave rest with defaults? Like with .conf files 2. what about Horizon and visibility of options? In mentioned case putting the same policy.json file in /etc/kolla/config/keystone/ and /etc/kolla/config/horizon/ should „unblock” Add User button for user with project_admin role? Or how to achieve it? 3. does Horizon need the duplicated policy.json files from other services in it’s configuration folder or is it enough to write policy.json for services I want to change? 4. when I assign admin role to a user with projectID (openstack role add —project PROJECT_ID —user SOME_USER admin) this user sees in Horizon everything systemwide, not only inside this project… Which rules should be created to allow him to see only users and resources which belongs to this project? Best regards Adam

2 7

[manila][ptg] Xena PTG Planning
by Goutham Pacha Ravi 15 Apr '21

15 Apr '21

Hello Zorillas and Interested Stackers, As you're aware, the virtual PTG for the Xena release cycle is between April 19-23, 2021. If you haven't registered yet, you must do so as soon as possible! [1]. We've signed up for some slots on the PTG timeslots ethercalc [2]. The PTG Planning etherpad [3] is now live. Please go ahead and add your name/irc nick and propose any topics. You may propose topics even if you wouldn't like to moderate the discussion. Thanks, and hope to see you all there! Goutham [1] https://april2021-ptg.eventbrite.com/ [2] https://ethercalc.net/oz7q0gds9zfi [3] https://etherpad.opendev.org/p/xena-ptg-manila-planning

1 1

[oslo][security-sig] Please revisit your open vulnerability report
by Jeremy Stanley 12 Apr '21

12 Apr '21

Please help the OpenStack Vulnerability Management Team by taking a look at the following report: keystonemiddleware connections to memcached from neutron-server grow beyond configured values https://launchpad.net/bugs/1883659 Can it be exploited by a nefarious actor, and if so, how? Is it likely to be fixable in all our supported stable branches, respecting stable backport policy? What deployment configurations and options might determine whether a particular installation is susceptible? This is the sort of feedback we depend on to make determinations regarding whether and how to keep the public notified, so they can make informed decisions. Thanks for doing your part to keep our users safe! -- Jeremy Stanley

2 6

[ops][cinder][nova] os-brick upcoming releases
by Brian Rosmaita 12 Apr '21

12 Apr '21

Hello operators, You may have heard about a potential data-loss bug [0] that was recently discovered. It has been fixed in the upcoming wallaby release and we are planning to backport to all stable branches and do new os-brick releases from the releasable stable branches. In the meantime, the bug occurs if the multipath configuration option on a compute is changed while volumes are attached to instances on that compute. The possible data loss may occur when the volumes are detached (migration, volume-detach, etc.). Thus, before the new os-brick releases are available, the issue can nonetheless be averted by not making such a configuration change under those circumstances. The new os-brick releases will be: - victoria: 4.0.3 - ussuri: 3.0.6 - train: 2.10.6 The stein, rocky, and queens branches are in Extended Maintenance mode and are no longer released from, but critical fixes are backported to them when possible, though it may take a while before these are merged. [0] https://launchpad.net/bugs/1921381

1 1

[dev][cinder][keystone] Properly consuming system-scope in cinder
by Lance Bragstad 09 Apr '21

09 Apr '21

Hey folks, As I'm sure some of the cinder folks are aware, I'm updating cinder policies to include support for some default personas keystone ships with. Some of those personas use system-scope (e.g., system-reader and system-admin) and I've already proposed a series of patches that describe what those changes look like from a policy perspective [0]. The question now is how we test those changes. To help guide that decision, I worked on three different testing approaches. The first was to continue testing policy using unit tests in cinder with mocked context objects. The second was to use DDT with keystonemiddleware mocked to remove a dependency on keystone. The third also used DDT, but included changes to update NoAuthMiddleware so that it wasn't as opinionated about authentication or authorization. I brought each approach in the cinder meeting this week where we discussed a fourth approach, doing everything in tempest. I summarized all of this in an etherpad [1] Up to yesterday morning, the only approach I hadn't tinkered with manually was tempest. I spent some time today figuring that out, resulting in a patch to cinderlib [2] to enable a protection test job, and cinder_tempest_plugin [3] that adds the plumbing and some example tests. In the process of implementing support for tempest testing, I noticed that service catalogs for system-scoped tokens don't contain cinder endpoints [4]. This is because the cinder endpoint contains endpoint templating in the URL [5], which keystone will substitute with the project ID of the token, if and only if the catalog is built for a project-scoped token. System and domain-scoped tokens do not have a reasonable project ID to use in this case, so the templating is skipped, resulting in a cinder service in the catalog without endpoints [6]. This cascades in the client, specifically tempest's volume client, because it can't find a suitable endpoint for request to the volume service [7]. Initially, my testing approaches were to provide examples for cinder developers to assess the viability of each approach before committing to a protection testing strategy. But, the tempest approach highlighted a larger issue for how we integrate system-scope support into cinder because of the assumption there will always be a project ID in the path (for the majority of the cinder API). I can think of two ways to approach the problem, but I'm hoping others have more. First, we remove project IDs from cinder's API path. This would be similar to how nova (and I assume other services) moved away from project-specific URLs (e.g., /v3/%{project_id}s/volumes would become /v3/volumes). This would obviously require refactoring to remove any assumptions cinder has about project IDs being supplied on the request path. But, this would force all authorization information to come from the context object. Once a deployer removes the endpoint URL templating, the endpoints will populate in the cinder entry of the service catalog. Brian's been helping me understand this and we're unsure if this is something we could even do with a microversion. I think nova did it moving from /v2/ to /v2.0/, which was technically classified as a major bump? This feels like a moon shot. Second, we update cinder's clients, including tempest, to put the project ID on the URL. After we update the clients to append the project ID for cinder endpoints, we should be able to remove the URL templating in keystone, allowing cinder endpoints to appear in system-scoped service catalogs (just like the first approach). Clients can use the base URL from the catalog and append the admin project ID before putting the request on the wire. Even though the request has a project ID in the path, cinder would ignore it for system-specific APIs. This is already true for users with an admin role on a project because cinder will allow you to get volumes in one project if you have a token scoped to another with the admin role [8]. One potential side-effect is that cinder clients would need *a* project ID to build a request, potentially requiring another roundtrip to keystone. Thoughts? [0] https://review.opendev.org/q/project:openstack/cinder+topic:secure-rbac [1] https://etherpad.opendev.org/p/cinder-secure-rbac-protection-testing [2] https://review.opendev.org/c/openstack/cinderlib/+/772770 [3] https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/772915 [4] http://paste.openstack.org/show/802117/ [5] http://paste.openstack.org/show/802097/ [6] https://opendev.org/openstack/keystone/src/commit/c239cc66615b41a0c09e031b3… [7] http://paste.openstack.org/show/802092/ [8] http://paste.openstack.org/show/802118/

4 8

[puppet] Proposing Alan Bishop (abishop) for puppet-cinder core and puppet-glance core
by Takashi Kajinami 08 Apr '21

08 Apr '21

Hello, I'd like to propose Alan Bishop (abishop) for the core team of puppet-cinder and puppet-glance. Alan has been actively involved in these 2 modules for a few years and has implemented some nice features like multiple backend support in glance, cinder s3 backup driver and etc, which expanded adoption of puppet-openstack. He has also provided good reviews on patches for these 2 repos based on his understanding about our code, puppet and serverspec. He is an active contributor to cinder and has deep knowledge about it. In addition He is also a core review in TripleO, which consumes our puppet modules, and mainly covers storage components like cinder and glance, so he is familiar with the way how these two components are deployed and configured. I believe adding him to our board helps us improve our review of these two modules. I'll wait for one week to hear any feedback from other core reviewers. Thank you, Takashi

6 6

[cinder] Review of tiny patch to add Ceph RBD fast-diff to cinder-backup
by Christian Rohmann 07 Apr '21

07 Apr '21

Hey everyone, I wrote a tiny patch to add the Ceph RDB feature of fast-diff to backups created by cinder-backup: * https://review.opendev.org/c/openstack/cinder/+/766856/ Could someone please take a peek and let me know of this is sufficient to be merged? Thanks and with kind regards, Christian

3 12

[docs] Project guides in PDF format
by Peter Matulis 07 Apr '21

07 Apr '21

Hello, I understand that there was a push [1] to make PDFs available for download from the published pages on docs.openstack.org for the various guides (Sphinx projects). Is there any steam left in this initiative? The next best thing is to have something at the repository level that people can work with. Currently, there are a limited number of guides that can be converted to PDF from the openstack-manuals repository. What is the suggested implementation for individual projects? Thanks, Peter Matulis [1]: https://etherpad.opendev.org/p/train-pdf-support-goal

5 11

[qa][hacking] Proposing new core reviewers
by Martin Kopec 06 Apr '21

06 Apr '21

Hi all, I'd like to propose Sorin Sbarnea (IRC: zbr) and Radosław Piliszek (IRC: yoctozepto) to hacking core. They both are doing a great upstream work among multiple different projects and volunteered to help us with maintenance of hacking project as well. You can vote/feedback in this email thread. If no objection by 6th of April, we will add them to the list. Regards, -- Martin Kopec

3 3

[all] Gate resources and performance
by Dan Smith 06 Apr '21

06 Apr '21

Hi all, I have become increasingly concerned with CI performance lately, and have been raising those concerns with various people. Most specifically, I'm worried about our turnaround time or "time to get a result", which has been creeping up lately. Right after the beginning of the year, we had a really bad week where the turnaround time was well over 24 hours. That means if you submit a patch on Tuesday afternoon, you might not get a test result until Thursday. That is, IMHO, a real problem and massively hurts our ability to quickly merge priority fixes as well as just general velocity and morale. If people won't review my code until they see a +1 from Zuul, and that is two days after I submitted it, that's bad. Things have gotten a little better since that week, due in part to getting past a rush of new year submissions (we think) and also due to some job trimming in various places (thanks Neutron!). However, things are still not great. Being in almost the last timezone of the day, the queue is usually so full when I wake up that it's quite often I don't get to see a result before I stop working that day. I would like to ask that projects review their jobs for places where they can cut out redundancy, as well as turn their eyes towards optimizations that can be made. I've been looking at both Nova and Glance jobs and have found some things I think we can do less of. I also wanted to get an idea of who is "using too much" in the way of resources, so I've been working on trying to characterize the weight of the jobs we run for a project, based on the number of worker nodes required to run all the jobs, as well as the wall clock time of how long we tie those up. The results are interesting, I think, and may help us to identify where we see some gains. The idea here is to figure out[1] how many "node hours" it takes to run all the normal jobs on a Nova patch compared to, say, a Neutron one. If the jobs were totally serialized, this is the number of hours a single computer (of the size of a CI worker) would take to do all that work. If the number is 24 hours, that means a single computer could only check *one* patch in a day, running around the clock. I chose the top five projects in terms of usage[2] to report here, as they represent 70% of the total amount of resources consumed. The next five only add up to 13%, so the "top five" seems like a good target group. Here are the results, in order of total consumption: Project % of total Node Hours Nodes ------------------------------------------ 1. TripleO 38% 31 hours 20 2. Neutron 13% 38 hours 32 3. Nova 9% 21 hours 25 4. Kolla 5% 12 hours 18 5. OSA 5% 22 hours 17 What that means is that a single computer (of the size of a CI worker) couldn't even process the jobs required to run on a single patch for Neutron or TripleO in a 24-hour period. Now, we have lots of workers in the gate, of course, but there is also other potential overhead involved in that parallelism, like waiting for nodes to be available for dependent jobs. And of course, we'd like to be able to check more than patch per day. Most projects have smaller gate job sets than check, but assuming they are equivalent, a Neutron patch from submission to commit would undergo 76 hours of testing, not including revisions and not including rechecks. That's an enormous amount of time and resource for a single patch! Now, obviously nobody wants to run fewer tests on patches before they land, and I'm not really suggesting that we take that approach necessarily. However, I think there are probably a lot of places that we can cut down the amount of *work* we do. Some ways to do this are: 1. Evaluate whether or not you need to run all of tempest on two configurations of a devstack on each patch. Maybe having a stripped-down tempest (like just smoke) to run on unique configs, or even specific tests. 2. Revisit your "irrelevant_files" lists to see where you might be able to avoid running heavy jobs on patches that only touch something small. 3. Consider moving some jobs to the experimental queue and run them on-demand for patches that touch particular subsystems or affect particular configurations. 4. Consider some periodic testing for things that maybe don't need to run on every single patch. 5. Re-examine tests that take a long time to run to see if something can be done to make them more efficient. 6. Consider performance improvements in the actual server projects, which also benefits the users. If you're a project that is not in the top ten then your job configuration probably doesn't matter that much, since your usage is dwarfed by the heavy projects. If the heavy projects would consider making changes to decrease their workload, even small gains have the ability to multiply into noticeable improvement. The higher you are on the above list, the more impact a small change will have on the overall picture. Also, thanks to Neutron and TripleO, both of which have already addressed this in some respect, and have other changes on the horizon. Thanks for listening! --Dan 1: https://gist.github.com/kk7ds/5edbfacb2a341bb18df8f8f32d01b37c 2; http://paste.openstack.org/show/C4pwUpdgwUDrpW6V6vnC/

12 36

[kolla][glance] Few questions about images.
by Eddie Yen 06 Apr '21

06 Apr '21

Hi everyone, I want to ask about the image operating permission & Windows images issue we met since we can't find any answers on the internet. 1. Now we're still using Rocky with Ceph as storage. Sometimes we need to re-pack the image on the Openstack. We used to save as snapshot (re-pack by Nova ephemeral VM) or upload to image (re-pack by volume), then set snapshot/image's visibility to the public. But since Rocky, we can't do this anymore because when we try to set public, Horizon always shows "not enough permission" error. The workaround we're using for now is creating a nova snapshot after re-pack, download the snapshot, then upload the snapshot again as public. But it's utterly wasting time if the images are huge. So we want to know how to unleash this limitation can let us just change snapshot to public at least. 2. Openstack uses virtio as a network device by default, so we always install a virtio driver when packing Windows images. As the network performance issue in GSO/TSO enablement, we also need to disable them in device properties. But since Windows 10 2004 build (my thought) device properties always reset these settings after Sysprep. We found there's a workaround [1] to solve this issue, but may not work sometimes. Is there a better way to solve this issue? Many thanks, Eddie. [1] PersistAllDeviceInstalls | Microsoft Docs <https://docs.microsoft.com/en-us/windows-hardware/customize/desktop/unatten…>

1 1

launch VM on volume vs. image
by Tony Liu 03 Apr '21

03 Apr '21

Hi, With Ceph as the backend storage, launching a VM on volume takes much longer than launching on image. Why is that? Could anyone elaborate the high level workflow for those two cases? Thanks! Tony

3 9

[ironic] Review Jams
by Julia Kreger 01 Apr '21

01 Apr '21

In the Ironic team's recent mid-cycle call, we discussed the need to return to occasionally having review jams in order to help streamline the review process. In other words, get eyes on a change in parallel and be able to discuss the change. The goal is to help get people on the same page in terms of what and why. Be on hand to answer questions or back-fill context. This is to hopefully avoid the more iterative back and forth nature of code review, which can draw out a long chain of patches. As always, the goal is not perfection, but forward movement especially for complex changes. We've established two time windows that will hopefully not to be too hard for some contributors to make it to. It doesn't need to be everyone, but it would help for at least some people whom actively review or want to actively participate in reviewing, or whom are even interested in a feature to join us for our meeting. I've added an entry on to our wiki page to cover this, with the current agenda and anticipated review jam topic schedule. The tl;dr is we will use meetpad[1] and meet on Mondays at 2 PM UTC and Tuesdays at 6 PM UTC. The hope is to to enable some overlap of reviewers. If people are interested in other times, please bring this up in the weekly meeting or on the mailing list. I'm not sending out calendar invites for this. Yet. :) See everyone next week! -Julia [0]: https://wiki.openstack.org/wiki/Meetings/Ironic#Review_Jams [1]: https://meetpad.opendev.org/ironic

3 2

How to customize the xml used in libvirt from GUI/opestack command line?
by Evan Zhao 01 Apr '21

01 Apr '21

Hi there, I googled this question and found two major answers: 1. ssh to the compute node and use `virsh dumpxml` and `virsh undefine/define`, etc. 2. edit nova/virt/libvrit/config.py directly. However, it's trivial to ssh to each node and do the modification, and I prefer not to touch the nova source code, is there any better ways to achieve this? I expect to edit the namespace of a certain element and append an additional element <qemu:command> to the xml file. Any information will be appreciated.

2 1

[cinder][nova][requirements] RFE requested for os-brick
by Brian Rosmaita 01 Apr '21

01 Apr '21

Hello Requirements Team, The Cinder team recently became aware of a potential data-loss bug [0] that has been fixed in os-brick master [1] and backported to os-brick stable/wallaby [2]. We've proposed a release of os-brick 4.4.0 from stable/wallaby [3] and are petitioning for an RFE to include 4.4.0 in the wallaby release. We have three jobs running tempest with os-brick source in master that have passed with [1]: os-brick-src-devstack-plugin-ceph [4], os-brick-src-tempest-lvm-lio-barbican [5],and os-brick-src-tempest-nfs [6]. The difference between os-brick master (at the time the tests were run) and stable/wallaby since the 4.3.0 tag is as follows: master: d4205bd 3 days ago iSCSI: Fix flushing after multipath cfg change (Gorka Eguileor) 0e63fe8 2 weeks ago Merge "RBD: catch read exceptions prior to modifying offset" (Zuul) 28545c7 4 months ago RBD: catch read exceptions prior to modifying offset (Jon Bernard) 99b2c60 2 weeks ago Merge "Dropping explicit unicode literal" (Zuul) 7cfdb76 6 weeks ago Dropping explicit unicode literal (tushargite96) 9afa1a0 3 weeks ago Add Python3 xena unit tests (OpenStack Release Bot) ab57392 3 weeks ago Update master for stable/wallaby (OpenStack Release Bot) 91a1cca 3 weeks ago (tag: 4.3.0) Merge "NVMeOF connector driver connection information compatibility fix" (Zuul) stable/wallaby: f86944b 3 days ago Add release note prelude for os-brick 4.4.0 (Brian Rosmaita) c70d70b 3 days ago iSCSI: Fix flushing after multipath cfg change (Gorka Eguileor) 6649b8d 3 weeks ago Update TOX_CONSTRAINTS_FILE for stable/wallaby (OpenStack Release Bot) f3f93dc 3 weeks ago Update .gitreview for stable/wallaby (OpenStack Release Bot) 91a1cca 3 weeks ago (tag: 4.3.0) Merge "NVMeOF connector driver connection information compatibility fix" (Zuul) This gives us very high confidence that the results of the tests run against master also apply to stable/wallaby at f86944b. Thank you for considering this request. (I've included Nova here because the bug occurs when the configuration option that enables multipath connections on a compute is changed while volumes are attached, so if this RFE is approved, nova might want to raise the minimum version of os-brick in wallaby to 4.4.0.) [0] https://launchpad.net/bugs/1921381 [1] https://review.opendev.org/c/openstack/os-brick/+/782992 [2] https://review.opendev.org/c/openstack/os-brick/+/783207 [3] https://review.opendev.org/c/openstack/releases/+/783641 [4] https://zuul.opendev.org/t/openstack/build/30a103668e4c4a8cb6f1ef907ef3edcb [5] https://zuul.opendev.org/t/openstack/build/bb11eef737d34c41bb4a52f8433850b0 [6] https://zuul.opendev.org/t/openstack/build/3ad3359ca712432d9ef4261d72c787fa

4 9

[tc] Technical Committee weekly meeting
by Ghanshyam Mann 31 Mar '21

31 Mar '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for April 1st at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, March 31st, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[ironic] PTL on vacation - Week of April 12th
by Julia Kreger 31 Mar '21

31 Mar '21

Greetings folks! I'm going to take a little time off before we get to the Project Teams Gathering, so aside from the two calls I've committed myself to during that week, I'm going to be sitting on a mountain enjoying a change of scenery. If someone will volunteer to run the weekly meeting during the week of the 12th, I will be greatly appreciative. I'm hoping to have a rough PTG schedule up in advance of that meeting as well as the topic list refined, so it will just be links in the weekly agenda. Thanks, -Julia

2 1

[neutron] oslo.privsep migration in Neutron
by Rodolfo Alonso Hernandez 31 Mar '21

31 Mar '21

Hello Neutrinos: During the last cycles we have been migrating the Neutron code from oslo.rootwrap to oslo.privsep. Those efforts are aimed at reaching the goal defined in [1] and are tracked in [2]. At this point, starting Xena developing cycle, we can state that we have migrated all short lived commands from oslo.rootwrap to oslo.privsep or to a native implementation (that could also use oslo.privsep to elevate the permissions if needed). The problem are the daemons or services (long lived processes) that Neutron spawns using "ProcessManager"; this is why "ProcessManager.enable" is the only code calling "utils.execute" without "privsep_exec" parameter. Those process cannot be executed using oslo.privsep because the privsep root daemon has a limited number of executing threads. The remaining processes are [3]. Although we didn't reach the Completion Criteria defined in [1], that is remove the oslo.rootwrap dependency, I think we don't have an alternative to run those services and we should keep rootwrap for them. If there are no objections, once [3] is merged we can consider that Neutron (not other Stadium projects) finished the efforts on [1]. Please, any feedback is always welcome. Regards. [1]https://review.opendev.org/c/openstack/governance/+/718177 [2]https://storyboard.openstack.org/#!/story/2007686 [3] https://review.opendev.org/c/openstack/neutron/+/778444/2/etc/neutron/rootw…

5 6

[ironic] How to move nodes from a 'clean failed' state into 'Available'
by Igal Katzir 31 Mar '21

31 Mar '21

Hello Team, I had a situation where my undercloud-node had a problem with it’s disk and has disconnected from overcloud. I couldn’t restore the undercloud controller and ended up re-installing it (running 'openstack undercloud install’). The installation ended successfully but now I’m in a situation where Cleanup of the overcloud deployed nodes fails: (undercloud) [stack@interop010 ~]$ openstack baremetal node list +--------------------------------------+------------+---------------+-------------+--------------------+-------------+ | UUID | Name | Instance UUID | Power State | Provisioning State | Maintenance | +--------------------------------------+------------+---------------+-------------+--------------------+-------------+ | 97b9a603-f64f-47c1-9fb4-6c68a5b38ff6 | interop025 | None | power on | clean failed | True | | 4b02703a-f765-4ebb-85ed-75e88b4cbea5 | interop026 | None | power on | clean failed | True | +--------------------------------------+------------+---------------+-------------+--------------------+-------------+ I’ve tried to move node to available state but cannot: (undercloud) [stack@interop010 ~]$ openstack baremetal node provide 97b9a603-f64f-47c1-9fb4-6c68a5b38ff6 The requested action "provide" can not be performed on node "97b9a603-f64f-47c1-9fb4-6c68a5b38ff6" while it is in state "clean failed". (HTTP 400) My question is: How do I make the nodes available again? as the deployment of overcloud fails with: ERROR due to "Message: No valid host was found. , Code: 500” Thanks, Igal

3 9

[ironic] PTG and topics oh my
by Julia Kreger 31 Mar '21

31 Mar '21

As everyone hopefully knows by now, the PTG[0] is coming up. We've had an etherpad[1] posted for about a month. During this time, the etherpad has collected 22 topics. I'm 99.95% sure there is some topic overlap, but what we need to make a schedule... or at least begin to form an agenda which requires us understanding what community members perceive as priorities or important topics. So if everyone over the next couple days could do the following two tasks, it would help tremendously for consensus building. 1) Add any missing topics you feel to be important 2) Add +1's to topics you feel are important for yourself OR the community. As always, pressing topics and topics which have the most votes will tend to have more time allocated. -Julia [0]: https://www.eventbrite.com/e/project-teams-gathering-april-2021-tickets-143… [1]: https://etherpad.opendev.org/p/ironic-xena-ptg

1 0

How to debug silent live migration errors
by Linebarger, John 31 Mar '21

31 Mar '21

How would I debug silent (or mostly silent) live migration errors? We're using the Stein release of Canonical's Charmed OpenStack. I have configured it for live migration per the instructions at this link: https://docs.openstack.org/nova/pike/admin/configuring-migrations.html#sect… Specifically: 1. I did not specify vncserver_listen=0.0.0.0 in nova.conf because we are not running VNC on our instances 2. instances_path is /var/lib/nova/instances on all compute nodes 3. I believe that MAAS is "the sole provider of DHCP and DNS for the network hosting the MAAS cluster", per https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/late… 4. Identical authorized_keys files are present on all compute nodes with keys from all compute nodes by default 5. I manually configured the firewalls on all compute nodes to allow libvirt to communicate between compute hosts with: sudo ufw allow 49152:49261/tcp 6. The following settings are specified in nova.conf on each compute node: live_migration_downtime = 500 live_migration_downtime_steps = 10 live_migration_downtime_delay = 75 live_migration_permit_post_copy=true Here's what happens when I try to Live Migrate from the Horizon Dashboard: 1. As admin, in the Admin --> Instances menu, I select the dropdown arrow to the right of the instance. Live Migrate Instance appears (but in black, unlike Migrate Instance, which appears in red). I select Live Migrate Instance, and whether or not I Automatically schedule new host or manually select a new host the Task column says "Migrating" and then it stops and reverts to None. The server never changes. The Action Log shows the live migration request but the Message column is blank. 2. I do the very same thing but this time select Disk Over Commit. Same results. Migrating reverts back to None and the server never changes. 3. I do the very same thing but this time select Block Migration. This time I do get an error: "Failed to live migrate instance to host 'AUTO_SCHEDULE'". And this time the Action Log has "Error" in the Message column. Same behavior with the CLI. For example, this CLI command below completes silently, yet the server for the instance never changes. john@vm-dev-john:~/bin$ openstack server migrate <instanceID> --live <newServerName> [Silent failure] john@vm-dev-john:~/bin$ openstack server show <instancID> [Still running on original server] Note that I *can* successfully Migrate, both using the Horizon Dashboard and the CLI. What fails is Live Migration. I just have no idea why, and no error is displayed in the Action Log for the instance. For reference, the instance is an m1.small with 2GB of RAM, 1 VCPU, and a 20GB Cinder disk volume attached on /dev/vda. Any and all debugging ideas would be most welcome. Without logs I am simply guessing in the dark at this point. Thanks! Enjoy! John M. Linebarger, PhD, MBA Principal Member of Technical Staff Sandia National Laboratories (Office) 505-845-8282 (Cell) 505-681-4879 [cid:image002.jpg@01D72535.958BCAE0]<http://www.sandia.gov/>[AWS Certified Solutions Architect - Professional]<https://www.youracclaim.com/badges/24fe4e43-2f72-4ecc-a11a-783d21dada0f>[AWS Certified Solutions Architect - Associate]<https://www.youracclaim.com/badges/707a5f6f-d919-4daa-bbc4-81b5a779a6cd>[AWS Certified Developer - Associate]<https://www.youracclaim.com/badges/b52ac093-5415-4493-85c3-b77e055211f5>[cid:image003.png@01D72531.072F13F0]<https://www.youracclaim.com/badges/d6e5a453-0e61-4dbe-9716-6af51f364710/pub…>[cid:image005.png@01D72535.958BCAE0]<https://www.youracclaim.com/badges/035d4967-f77d-476e-915b-1061bb789ec3/pub…>

3 3

[cinder] Bug deputy report for week of 2021-03-31
by Sofia Enriquez 31 Mar '21

31 Mar '21

Hello, This is a bug report from 2021-03-16 to 2021-03-24. You're welcome to join the next Cinder Bug Meeting later today. Weekly on Wednesday at 1500 UTC in #openstack-cinder Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Critical: - https://bugs.launchpad.net/os-brick/+bug/1921381 "iSCSI: Flushing issues when multipath config has changed". Stable/ussuri backport assigned to Luigi Toscano (ltoscano) High: - Medium: - https://bugs.launchpad.net/cinder/+bug/1922013 "IBM SVF driver: Add vols to a group (consitent_group_replication enabled) fails sometimes if rccg and rcrel are in different states" Unassigned. Low: - https://bugs.launchpad.net/cinder/+bug/1921584: "backup/drivers/s3.py uses vendored requests from botocore". Unassigned. Incomplete: - Regards Sofi -- L. Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[kolla] does someone uses AArch64 images?
by Marcin Juszkiewicz 31 Mar '21

31 Mar '21

AArch64 support in Kolla(-Ansible) project is present since Ocata (iirc). For most of that time we supported all three distributions: CentOS, Debian and Ubuntu. # Distribution coverage CentOS has binary (with packages from RDO project) and source images. We supported version 7, 8 and now we are using CentOS Stream 8 in Wallaby. Debian is source only as there never was any source of up-to-date OpenStack packages (Debian OpenStack team builds only x86-64 packages and they do it after release). Ubuntu has both binary (packages from UCA) and source images. # My interest I take care of Debian/source ones as we use them at Linaro in our developer cloud setups. It often means backporting of fixes when we upgrade from one previous release to something newer. # External repos problem The problem in all cases is external repositories and their lack of support for AArch64 architecture. ## RabbitMQ Biggest problem is with RabbitMQ and Erlang. For x86-64 RMQ developers build packages for CentOS/Debian/Ubuntu. We use them. On AArch64 it differ: - CentOS uses erlang from CentOS distribution - Debian uses erlang from my Linaro OBS repository (where I do builds of backport of Debian 'testing' package). - Ubuntu uses erlang from Ubuntu distribution Ubuntu one is too old - 22.2 while current rabbitmq needs 22.3 or newer. I do not plan to work on getting it working. # Final question I would like to know does anyone use AArch64 images built with Kolla. Please respond which branches you use, distribution used as host OS and as images OS.

3 9

RE: Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 31 Mar '21

31 Mar '21

Centos7: {rabbit,"RabbitMQ","3.7.24"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, Centos8: {rabbit,"RabbitMQ","3.7.28"}, "Erlang/OTP 22 [erts-10.7.2.8] [source] [64-bit] [smp:1:1] [ds:1:1:10] [async-threads:128] [hipe]\n"}, When I deploy the first Centos8 controller, RMQ comes up with all 3 nodes active and seems to be working fine until I shut down the 2nd controller. The only hint of trouble when I replace the 1st node is this error message the first time I run the deployment: https://paste.ubuntu.com/p/h9HWdfwmrK/ and the crash dump that appears on control2: crash dump log: https://paste.ubuntu.com/p/MpZ8SwTJ2T/ First 1500 lines of the dump: https://paste.ubuntu.com/p/xkCyp2B8j8/ If I wait for a few minutes then RMQ recovers on control2 and the 2nd run of the deployment seems to work, and there is no trouble until I shut down control1. -----Original Message----- From: Mark Goddard <mark(a)stackhpc.com> Sent: Wednesday, March 31, 2021 4:14 AM To: Braden, Albert <C-Albert.Braden(a)charter.com> Cc: openstack-discuss(a)lists.openstack.org Subject: [EXTERNAL] Re: [kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Tue, 30 Mar 2021 at 13:41, Braden, Albert <C-Albert.Braden(a)charter.com> wrote: > > I’ve created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: > > > > https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… > > > > I used the instructions here to successfully remove and replace control0 with a Centos8 box > > > > https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… > > > > After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, > > {'rabbit@chrnc-void-testupgrade-control-1',[]}, > > {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] > > > > After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: > > > > kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 > > … > > control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 > > > > After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. > > > > If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: > > > > (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status > > Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... > > [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', > > 'rabbit@chrnc-void-testupgrade-control-0-replace', > > 'rabbit@chrnc-void-testupgrade-control-1', > > 'rabbit@chrnc-void-testupgrade-control-2']}]}, > > {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', > > 'rabbit@chrnc-void-testupgrade-control-0-replace']}, > > {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, > > {partitions,[]}, > > {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, > > {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] > > > > But my hypervisors are down: > > > > (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | > > | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | > > | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | > > +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ > > > > When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: > > > > 172.16.2.31 compute0 > > 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. > > 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out > > > > In the RMQ logs I see this every 10 seconds: > > > > 172.16.1.132 control2 > > [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 > > 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > client unexpectedly closed TCP connection > > 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) > > 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e > > 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' > > 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): > > > > Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? Hi Albert, Could you share the versions of RabbitMQ and erlang in both versions of the container? When initially testing this setup, I think we had 3.7.24 on both sides. Perhaps the CentOS 8 version has moved on sufficiently to become incompatible? Mark > > > > I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. > > > > The contents of this e-mail message and > any attachments are intended solely for the > addressee(s) and may contain confidential > and/or legally privileged information. If you > are not the intended recipient of this message > or if this message has been addressed to you > in error, please immediately alert the sender > by reply e-mail and then delete this message > and any attachments. If you are not the > intended recipient, you are notified that > any use, dissemination, distribution, copying, > or storage of this message or any attachment > is strictly prohibited. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

1 0

[qa][tempest] Announcing scenario.manager stable interface
by Martin Kopec 31 Mar '21

31 Mar '21

Hi all, we would like to announce that tempest.scenario.manager has been declared stable in Tempest 27.0.0 and it's ready to be consumed by tempest plugins. All related patches were tracked under 'tempest-scenario-manager-stable' blueprint [1] and the whole effort was tracked in [2]. A little history: Some time ago, tempest/scenario/manager.py got copied to most of the plugins and therefore, it diverged - every plugin's copy had slight differences. In the latest release, we pushed changes to unify the manager's methods and improved their APIs in order to have them easier to consume. I would also like to thank Soniya Vyas for an exceptional job moving this effort forward. [1] https://review.opendev.org/q/topic:bp/tempest-scenario-manager-stable [2] https://etherpad.opendev.org/p/tempest-scenario-manager Regards, -- Martin Kopec

5 4

[Swift] swift client querying capabilites fails and may affect Horizon's container management
by Stefan Kelber 31 Mar '21

31 Mar '21

Hello, i have a Kolla-Ansible AIO stack i added SWIFT to after the initial installation. CLI management of containers work on CLI, but Horizon is troubled: Clicking "container" throws "Error: Unable to fetch the policy details", and no horizon based container management is possible therefore. In an effort to find the root problem, i stumbled upon the following, which i assume to be related: it looks to me like there are issues with keystone authentication of SWIFT client (3.10.1) when attempting to fetch capabilities on OS Victoria Kolla-Ansible, but *only* when querying capabilities: <EXAMPLES> ## 1 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 stat Account: AUTH_4334f282aa3a498aac95d2cf9aa9fa91 Containers: 1 Objects: 2 Bytes: 0 Containers in policy "policy-0": 1 Objects in policy "policy-0": 2 Bytes in policy "policy-0": 0 Content-Type: text/plain; charset=utf-8 X-Timestamp: 1616964437.15820 Accept-Ranges: bytes X-Account-Project-Domain-Id: default Vary: Accept X-Trans-Id: tx2638af9bd98646ce92dfe-00606342a1 X-Openstack-Request-Id: tx2638af9bd98646ce92dfe-00606342a1 Transfer-Encoding: chunked ## 2 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 list mycontainer ## 3 (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 capabilities Capabilities GET failed: http://10.10.10.82:8080/info 401 Unauthorized [first 60 chars of response] b'{"error": {"code": 401, "title": "Unauthorized", "message": ' Failed Transaction ID: txeb57fccc09e54e3fa2fa5-00606342af </EXAMPLES> Am i on the wrong track? Best Stefan Here is the entire debug output of the failing transaction: (swift-container-server)[root@testhost ~]# swift --os-auth-url http://10.10.10.82:35357/v3 --auth-version 3 --os-project-name admin --os-project-domain-name Default --os-username admin --os-user-domain-name Default --os-password T-STRING-s6 capabilities --debug DEBUG:keystoneclient.auth.identity.v3.base:Making authentication request to http://10.10.10.82:35357/v3/auth/tokens DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.10.82:35357 DEBUG:urllib3.connectionpool:http://10.10.10.82:35357 "POST /v3/auth/tokens HTTP/1.1" 201 7299 DEBUG:keystoneclient.auth.identity.v3.base:{"token": {"methods": ["password"], "user": {"domain": {"id": "default", "name": "Default"}, "id": "12940206c0a1464f96a47fe74d19c893", "name": "admin", "password_expires_at": null}, "audit_ids": ["WjAnZyn5TaGufxZGqnvW3Q"], "expires_at": "2021-03-31T14:22:13.000000Z", "issued_at": "2021-03-30T14:22:13.000000Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "4334f282aa3a498aac95d2cf9aa9fa91", "name": "admin"}, "is_domain": false, "roles": [{"id": "b740e2667f034d02b6af2114ec06e20a", "name": "admin"}, {"id": "ceb68a7214c74a969cd83b9146e1fbda", "name": "heat_stack_owner"}, {"id": "6b2ae10793c34fb287dd994a8dd2e8a1", "name": "member"}, {"id": "edb76a75b2fc4af79bf09e38e07be5f1", "name": "reader"}], "catalog": [{"endpoints": [{"id": "05e0b6b67f094b64b1caa9cf78dd34b2", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}, {"id": "9575e60e3a034ff282bbeda2287b3d01", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}, {"id": "f5fb2fffde5d4b88b230397560ac6952", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8780", "region": "RegionOne"}], "id": "16edbecc93484568a2f5b46db9909293", "type": "placement", "name": "placement"}, {"endpoints": [{"id": "09275022c3de4405adb6bb25071a49ba", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}, {"id": "3a6fa026c7c9422f8cb013492f5ba0e0", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}, {"id": "a4fa0d097eed4c3b97ff4ed0f27096d6", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:9292", "region": "RegionOne"}], "id": "2a71f9f773734edb8ea6b711c844b334", "type": "image", "name": "glance"}, {"endpoints": [{"id": "a20fbce4737540ce91039175c7409811", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}, {"id": "b16bad7fdf1342329a59e231cf56851d", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}, {"id": "ce27e7239706458c80e42a68a13383b2", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2.1", "region": "RegionOne"}], "id": "2c56db642dde46a2ab2f766dbd0ba9c9", "type": "compute", "name": "nova"}, {"endpoints": [{"id": "8c183a6cd8244b2cb1630a662c3a292e", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "958487dfae194f69b5f56f0b6e901163", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a8e0db3b2e1c49a99ed97ec18a64ece6", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8004/v1/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "5428e2837ccf413886140c0603867ee3", "type": "orchestration", "name": "heat"}, {"endpoints": [{"id": "2e9a6a2fc78340b7bdc18041dec79a86", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "894ab56c6c7140ebaac8e7ec502e9b03", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "8e86543bfeb547d78cf707100d0ef7f0", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8774/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "5639441ae48441ed9c8163fc537f0c1a", "type": "compute_legacy", "name": "nova_legacy"}, {"endpoints": [{"id": "32092c9ac4634ece9d89edc2b146af54", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}, {"id": "dad19d0e0d8e420caa5b1276d82917e4", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}, {"id": "f085e59dae874005b8e6fb1df55db5d5", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:9696", "region": "RegionOne"}], "id": "56e63ef1e3b24fbca6a2efb7394eb77b", "type": "network", "name": "neutron"}, {"endpoints": [{"id": "0ccb4189837f482aac1ee67a0f5b69b7", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}, {"id": "be4fab7e4ae04786817702817ea09c34", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}, {"id": "eb8ccdc41ab94aaa9567e29e728e9109", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8000/v1", "region": "RegionOne"}], "id": "83cee8b745d74b1d9a3bc1e37d3e21bf", "type": "cloudformation", "name": "heat-cfn"}, {"endpoints": [{"id": "917c36b07fae4b278a2f63d3886933c3", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:5000", "region": "RegionOne"}, {"id": "9d40ecfea4a04327b64e5751ed8043ce", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:5000", "region": "RegionOne"}, {"id": "d4941b09a2cb40648f3b019bb1653120", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:35357", "region": "RegionOne"}], "id": "94dce13398fd42e4805ae1a7d9976e78", "type": "identity", "name": "keystone"}, {"endpoints": [{"id": "10297840af2b4370a2f63908aea3ff9c", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a1a617f8996d439b8de091a4f15a88ae", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "a930062cb2da497bbeb727c9fec6bba0", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v2/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "d7f1cd74b2294126ac939b7d705369a3", "type": "volumev2", "name": "cinderv2"}, {"endpoints": [{"id": "1bb18181af9b4dd28bbf16491adb16fe", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1/AUTH_4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "686b82631d4d40b1b62ed1a7cb8fb1cf", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1", "region": "RegionOne"}, {"id": "e0cc80d07c6246668ca9cce9d736af62", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8080/v1/AUTH_4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "eabcc4d34b3c4ce499025816e2302f34", "type": "object-store", "name": "swift"}, {"endpoints": [{"id": "4c2b1dbe8d3544c9add475d63b0492d4", "interface": "public", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "77634a04b3644d0eb2abab41f3325a1c", "interface": "internal", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}, {"id": "7f11d1c28644413e8cfe5ff19b91ebd2", "interface": "admin", "region_id": "RegionOne", "url": "http://10.10.10.82:8776/v3/4334f282aa3a498aac95d2cf9aa9fa91", "region": "RegionOne"}], "id": "fda50e1a62564367bca385a64088334d", "type": "volumev3", "name": "cinderv3"}]}} DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 10.10.10.82:8080 DEBUG:urllib3.connectionpool:http://10.10.10.82:8080 "GET /info HTTP/1.1" 401 114 INFO:swiftclient:REQ: curl -i http://10.10.10.82:8080/info -X GET -H "Accept-Encoding: gzip" INFO:swiftclient:RESP STATUS: 401 Unauthorized INFO:swiftclient:RESP HEADERS: {'Content-Type': 'application/json', 'Content-Length': '114', 'WWW-Authenticate': 'Keystone uri="http://10.10.10.82:5000"', 'X-Trans-Id': 'txac041bfff30342729c2dd-0060633415', 'X-Openstack-Request-Id': 'txac041bfff30342729c2dd-0060633415', 'Date': 'Tue, 30 Mar 2021 14:22:13 GMT'} INFO:swiftclient:RESP BODY: b'{"error": {"code": 401, "title": "Unauthorized", "message": "The request you have made requires authentication."}}' Capabilities GET failed: http://10.10.10.82:8080/info 401 Unauthorized [first 60 chars of response] b'{"error": {"code": 401, "title": "Unauthorized", "message": ' Failed Transaction ID: txac041bfff30342729c2dd-0060633415

1 1

[kolla] Train Centos7 -> Centos8 upgrade fails on 2nd controller
by Braden, Albert 31 Mar '21

31 Mar '21

I've created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… I used the instructions here to successfully remove and replace control0 with a Centos8 box https://docs.openstack.org/kolla-ansible/train/user/adding-and-removing-hos… After this my RMQ admin page shows all 3 nodes up, including the new control0. The name of the cluster is rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com<mailto:rabbit@chrnc-void-testupgrade-control-0.dev.chtrse.com> (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-0-replace',[]}, {'rabbit@chrnc-void-testupgrade-control-1',[]}, {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] After that I create a new VM to verify that the cluster is still working, and then perform the same procedure on control1. When I shut down services on control1, the ansible playbook finishes successfully: kolla-ansible -i ../multinode stop --yes-i-really-really-mean-it --limit control1 ... control1 : ok=45 changed=22 unreachable=0 failed=0 skipped=105 rescued=0 ignored=0 After this my RMQ admin page stops responding. When I check RMQ on the new control0 and the existing control2, the container is still up but RMQ is not running: (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status Error: this command requires the 'rabbit' app to be running on the target node. Start it with 'rabbitmqctl start_app'. If I start it on control0 and control2, then the cluster seems normal and the admin page starts working again, and cluster status looks normal: (rabbitmq)[root@chrnc-void-testupgrade-control-0-replace /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-0-replace ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-2', 'rabbit@chrnc-void-testupgrade-control-0-replace']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-2',[]}, {'rabbit@chrnc-void-testupgrade-control-0-replace',[]}]}] But my hypervisors are down: (openstack) [root@chrnc-void-testupgrade-build kolla-ansible]# ohll +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ | 3 | chrnc-void-testupgrade-compute-2.dev.chtrse.com | QEMU | 172.16.2.106 | down | 5 | 8 | 2560 | 30719 | | 6 | chrnc-void-testupgrade-compute-0.dev.chtrse.com | QEMU | 172.16.2.31 | down | 5 | 8 | 2560 | 30719 | | 9 | chrnc-void-testupgrade-compute-1.dev.chtrse.com | QEMU | 172.16.0.30 | down | 5 | 8 | 2560 | 30719 | +----+-------------------------------------------------+-----------------+--------------+-------+------------+-------+----------------+-----------+ When I look at the nova-compute.log on a compute node, I see RMQ failures every 10 seconds: 172.16.2.31 compute0 2021-03-30 03:07:54.893 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out 2021-03-30 03:07:55.905 7 INFO oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] Reconnected to AMQP server on 172.16.1.132:5672 via [amqp] client with port 56422. 2021-03-30 03:08:05.915 7 ERROR oslo.messaging._drivers.impl_rabbit [req-70d69b45-c3a7-4fbc-b709-4d7d757e09e7 - - - - -] [aeb317a8-873f-49be-a2a0-c6d6e0891a3e] AMQP server on 172.16.1.132:5672 is unreachable: timed out. Trying again in 1 seconds.: timeout: timed out In the RMQ logs I see this every 10 seconds: 172.16.1.132 control2 [root@chrnc-void-testupgrade-control-2 ~]# tail -f /var/log/kolla/rabbitmq/rabbit\(a)chrnc-void-testupgrade-control-2.log |grep 172.16.2.31 2021-03-30 03:07:54.895 [warning] <0.13247.35> closing AMQP connection <0.13247.35> (172.16.2.31:56420 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): client unexpectedly closed TCP connection 2021-03-30 03:07:55.901 [info] <0.15288.35> accepting AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) 2021-03-30 03:07:55.903 [info] <0.15288.35> Connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672) has a client-provided name: nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e 2021-03-30 03:07:55.904 [info] <0.15288.35> connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e): user 'openstack' authenticated and granted access to vhost '/' 2021-03-30 03:08:05.916 [warning] <0.15288.35> closing AMQP connection <0.15288.35> (172.16.2.31:56422 -> 172.16.1.132:5672 - nova-compute:7:aeb317a8-873f-49be-a2a0-c6d6e0891a3e, vhost: '/', user: 'openstack'): Why does RMQ fail when I shut down the 2nd controller after successfully replacing the first one? I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 1

Networking-midonet deprecated
by Slawek Kaplonski 31 Mar '21

31 Mar '21

Hi, As it was discussed and decided in the other thread [1] due to lack of maintainers and not good shape of the project, networking-midonet was deprecated and is not part of the official Neutron stadium. Almost all required patches from [2] are already merged. Also patch [3] is merged already so there will be no official Wallaby release of the networking-midonet project. Last stable, supported release is Victroria. Project can be revived in the x/ namespace if there will be someone who wants to maintain it. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/021400.ht… [2] https://review.opendev.org/q/topic:%22deprecate-networking-midonet%22+(stat… [3] https://review.opendev.org/c/openstack/releases/+/781713 -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

How to customize the xml used in libvirt from GUI/opestack command line?
by Evan Zhao 30 Mar '21

30 Mar '21

Hi there, I googled this question and found two major answers: 1. ssh to the compute node and use `virsh dumpxml` and `virsh undefine/define`, etc. 2. edit nova/virt/libvrit/config.py directly. However, it's trivial to ssh to each node and do the modification, and I prefer not to touch the nova source code, is there any better ways to achieve this? I expect to edit the namespace of a certain element and append an additional element <qemu:command> to the xml file. Any information will be appreciated.

1 0

[security sig] April meeting cancelled
by Gage Hugo 30 Mar '21

30 Mar '21

Hey team, The security sig meeting for April 1st is cancelled, we will meet again next month at the usual time. Thanks

1 0

[security sig] PTG Timeslot
by Gage Hugo 30 Mar '21

30 Mar '21

Hello, The security SIG has a spot at the PTG from 1500 to 1700 UTC on April 19th. The agenda[0] is up for anyone to add if they have something to discuss. Hope to see you all there. [0] https://etherpad.opendev.org/p/security-sig-ptg-xena

1 0

[oslo] fix flake8-hacking inconsistences on xena/wallaby
by Herve Beraud 30 Mar '21

30 Mar '21

Hello Osloers, As you surely recently observed patches proposed by the openstack bot to configure wallaby fail with a flake8/hacking issue. Here are some guideline to fix the problem the inconsistency: - Patch pre-commit on master (now xena) (here is the patch: https://gist.github.com/4383/6e96c836bb1b0e1e0e599a5106f43f1a) - Once Xena is patched cherry-pick and backport these changes on Wallaby - Rebase the openstack bot patches on the top of this cherry-pick (or wait for the merge of the previous one patch) You can copy the patch on oslo.db with the related commit message [1]. The root cause of the issue was that with the introduction of pre-commit we started to define the version of flake8 to use. Previously this version was defined by hacking's requirements. Indeed a few months ago we added pre-commit to allow us to run checks with git hooks and reduce the usage of our gates. These changes were standardized and spread on all the scope of oslo [2]. However, during the design of these changes [3] and after some discussion we decided to pin the version of flake8 to use, hence by doing this we short circuited hacking on its management of flake8. The solution to solve this issue is simply to trust hacking on its flake8 management. Hacking will pull the right version of flake8 and the inconsistency will disappear. flake8 provides a pre-commit hook so it could be seen and called as a local target. [1] https://review.opendev.org/c/openstack/oslo.db/+/781470 [2] https://review.opendev.org/q/topic:%22oslo-pre-commit%22 [3] https://review.opendev.org/q/topic:%22pre-commit%22+(status:open%20OR%20sta… -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

4 7

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 30 Mar '21

30 Mar '21

> Oh, I have a guess what this might actually be. During snapshot upload process > user token that is used for the upload might get expired. If that's the case, > following changes in user_variables might help to resolve the issue: > > glance_glance_api_conf_overrides: > keystone_authtoken: > service_token_roles_required: True > service_token_roles: service I found out that after inserting the above the problems now occurs every time I try to take a snapshot of an instance. Below are my logs: For swift: Mar 29 12:50:37 infra1-swift-proxy-container-27169fa7 proxy-server[2049]: Could not autocreate account '/AUTH_024cc551782f41e395d3c9f13582ef7d' (txn: tx340e7873c3 ce42ab9808e-006061cd0f) Mar 29 12:50:37 infra1-swift-proxy-container-27169fa7 proxy-server[2049]: 192.168.110.106 192.168.110.211 29/Mar/2021/12/50/37 PUT /v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance_images HTTP/1.0 50 3 - python-swiftclient-3.10.1 gAAAAABgYczvODdX... - 118 - tx340e7873c3ce42ab9808e-006061cd0f - 14.3464 - - 1617022223.371025085 1617022237.717381477 - Mar 29 12:50:40 infra1-swift-proxy-container-27169fa7 proxy-server[2053]: ERROR with Account server 10.0.3.212:6002/os-objects re: Trying to HEAD /v1/AUTH_024cc551782f41e395d 3c9f13582ef7d: Host unreachable (txn: tx44e98ea667f04751bf22e-006061cd1f) Mar 29 12:50:40 infra1-swift-proxy-container-27169fa7 proxy-server[2053]: Account HEAD returning 503 for [] (txn: tx44e98ea667f04751bf22e-006061cd1f) Mar 29 12:50:40 infra1-swift-proxy-container-27169fa7 proxy-server[2053]: - - 29/Mar/2021/12/50/40 HEAD /v1/AUTH_024cc551782f41e395d3c9f13582ef7d%3Fformat%3Djson HTTP/1.0 503 - Swift - - - - tx44e98 ea667f04751bf22e-006061cd1f - 1.0482 RL - 1617022239.738376379 1617022240.786619186 - and for glance: Mar 29 12:50:56 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:50:56.155 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:50:56 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:50:56.156 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx44e98ea667f04751bf22e-006061cd1f', 'x-openstack-request-id': 'tx44e98ea667f04751bf22e-006061cd1f', 'date': 'Mon, 29 Mar 2021 12:50:56 GMT', 'connection': 'close'} Mar 29 12:50:56 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:50:56.157 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.544 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgYczvODdX..." -H "Content-Length: 0" Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.545 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.545 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx01d5fb0cb5cb4f6dbff12-006061cd34', 'x-openstack-request-id': 'tx01d5fb0cb5cb4f6dbff12-006061cd34', 'date': 'Mon, 29 Mar 2021 12:51:15 GMT', 'connection': 'close'} Mar 29 12:51:15 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:15.545 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.007 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgYczvODdX..." -H "Content-Length: 0" Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.008 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.009 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx9cab391639a344f4aa548-006061cd4b', 'x-openstack-request-id': 'tx9cab391639a344f4aa548-006061cd4b', 'date': 'Mon, 29 Mar 2021 12:51:42 GMT', 'connection': 'close'} Mar 29 12:51:42 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:51:42.009 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.472 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] REQ: curl -i http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… -X PUT -H "X-Auth-Token: gAAAAABgYczvODdX..." -H "Content-Length: 0" Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.473 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP STATUS: 503 Service Unavailable Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.473 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP HEADERS: {'content-type': 'text/html; charset=UTF-8', 'content-length': '118', 'x-trans-id': 'tx926ae448b316485d93a68-006061cd6e', 'x-openstack-request-id': 'tx926ae448b316485d93a68-006061cd6e', 'date': 'Mon, 29 Mar 2021 12:52:16 GMT', 'connection': 'close'} Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.474 2403 INFO swiftclient [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] RESP BODY: b'<html><h1>Service Unavailable</h1><p>The server is currently unavailable. Please try again at a later time.</p></html>' Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.476 2403 ERROR glance.api.v2.image_data [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Failed to upload image data due to internal error: glance_store.exceptions.BackendException: Failed to add container to Swift. Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Caught error: Failed to add container to Swift. Got error from Swift: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently'.: glance_store.exceptions.BackendException: Failed to add container to Swift. Got error from Swift: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently'. 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1154, in _create_container_if_missing 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi connection.head_container(container) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1870, in head_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return self._retry(None, head_container, container, headers=headers) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1801, in _retry 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi rv = func(self.url, self.token, *args, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1097, in head_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise ClientException.from_response( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: Container HEAD failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 404 Not Found 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1162, in _create_container_if_missing 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi connection.put_container(container) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1890, in put_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return self._retry(None, put_container, container, headers=headers, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1801, in _retry 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi rv = func(self.url, self.token, *args, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/swiftclient/client.py", line 1144, in put_container 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise ClientException.from_response(resp, 'Container PUT failed', body) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi swiftclient.exceptions.ClientException: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently' 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi During handling of the above exception, another exception occurred: 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Traceback (most recent call last): 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1347, in __call__ 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi action_result = self.dispatch(self.controller, action, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/wsgi.py", line 1391, in dispatch 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return method(*args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/common/utils.py", line 416, in wrapped 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return func(self, req, *args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 298, in upload 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self._restore(image_repo, image) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.force_reraise() 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise value 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/v2/image_data.py", line 163, in upload 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi image.set_data(data, size, backend=backend) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/domain/proxy.py", line 208, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.base.set_data(data, size, backend=backend, set_active=set_active) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 501, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi _send_notification(notify_error, 'image.upload', msg) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.force_reraise() 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi six.reraise(self.type_, self.value, self.tb) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise value 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/notifier.py", line 447, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.repo.set_data(data, size, backend=backend, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/api/policy.py", line 198, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return self.image.set_data(*args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/quota/__init__.py", line 318, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self.image.set_data(data, size=size, backend=backend, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 567, in set_data 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self._upload_to_store(data, verifier, backend, size) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance/location.py", line 458, in _upload_to_store 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi multihash, loc_meta) = self.store_api.add_with_multihash( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 398, in add_with_multihash 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return store_add_to_backend_with_multihash( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/multi_backend.py", line 480, in store_add_to_backend_with_multihash 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi (location, size, checksum, multihash, metadata) = store.add( 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/driver.py", line 279, in add_adapter 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi metadata_dict) = store_add_fun(*args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/capabilities.py", line 176, in op_checker 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi return store_op_fun(store, *args, **kwargs) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 942, in add 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi self._create_container_if_missing(location.container, 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi File "/openstack/venvs/glance-22.1.0/lib/python3.8/site-packages/glance_store/_drivers/swift/store.py", line 1167, in _create_container_if_missing 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi raise glance_store.BackendException(msg) 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi glance_store.exceptions.BackendException: Failed to add container to Swift. 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Got error from Swift: Container PUT failed: http://192.168.110.211:8080/v1/AUTH_024cc551782f41e395d3c9f13582ef7d/glance… 503 Service Unavailable [first 60 chars of response] b'<html><h1>Service Unavailable</h1><p>The server is currently'. 2021-03-29 12:52:16.552 2403 ERROR glance.common.wsgi Mar 29 12:52:16 infra1-glance-container-99614ac2 uwsgi[2403]: Mon Mar 29 12:52:16 2021 - uwsgi_response_writev_headers_and_body_do(): Connection reset by peer [core/writer.c line 306] during PUT /v2/images/15af8603-2428-4a9d-9202-eb7250cfea46/file (192.168.110.213) Mar 29 12:52:16 infra1-glance-container-99614ac2 glance-wsgi-api[2403]: 2021-03-29 12:52:16.565 2403 CRITICAL glance [req-d19ce0e3-2577-482e-ae09-9ea3c63aabe2 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unhandled error: OSError: write error 2021-03-29 12:52:16.565 2403 ERROR glance OSError: write error 2021-03-29 12:52:16.565 2403 ERROR glance Kind regards, Oliver

2 1

[charms] Charms libraries freeze on Friday
by Alex Kavanagh 30 Mar '21

30 Mar '21

Hi All Just a reminder that this week is the last opportunity to get features into the libraries for the release. The libraries affected by this freeze are: - charm-helpers - charms.openstack - charms.ceph - zaza - zaza-openstack-tests We will be taking a stable cut (stable/21.04) of these libraries on Friday and doing a batch review of the charms to point to these cuts during the testing period with a final charm-helpers sync at this point. The reactive charms will have a build.lock file introduced to lock their versions of the layers, interfaces and python modules (the above libraries will be tracked by stable branch). Only bug fixes and features-by-exception will be accepted into the stable cuts of these libraries after that point. For the avoidance of doubt, we will be testing and releasing against the current latest stable of: - juju 2.8.10 - python-libjuju 2.8.6 - charm-tools 2.8.3 Many thanks Alex. -- Alex Kavanagh - Software Engineer OpenStack Engineering - Data Centre Development - Canonical Ltd

1 0

[ops][victoria] Windows Server 2012 R2 - Can't install - unbootable
by DHilsbos＠performair.com 30 Mar '21

30 Mar '21

All; We have a recently built Victoria cluster with storage provided by a Ceph Nautilus cluster. I'm working through the following tutorial: https://medium.com/@yooniks9/openstack-install-windows-server-2016-2019-wit… When I get to step 4 under "Windows Server Installation," it says "Windows can't be installed on this drive." Details are: " Windows cannot be installed to this disk. This computer's hardware may not support booting to this disk. Ensure the disk's controller is enabled in the computer's BIOS menu. Any recommendations on what I'm missing here would be appreciated. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

3 2

[cinder] this week's meeting in video+IRC
by Brian Rosmaita 30 Mar '21

30 Mar '21

Quick reminder that this week's Cinder team meeting on Wednesday 31 March, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. Here's a quick reminder of the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. connection info: https://bluejeans.com/3228528973 cheers, brian

1 0

[neutron][db] Train: neutron-db-manage can't locate revision
by Eugen Block 30 Mar '21

30 Mar '21

Hi *, I have a new issue with our Train cloud (upgraded through the years). Currently I'm preparing the next upgrade step to Ussuri and stumbled upon this neutron db issue: ---snip--- controller01:~ # neutron-db-manage current Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. ERROR [alembic.util.messaging] Can't locate revision identified by 'e4e236b0e1ff' FAILED: Can't locate revision identified by 'e4e236b0e1ff' controller01:~ # neutron-db-manage revision --contract Running revision for neutron ... Generating /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/contract/b5344a66e818_.py ... done OK controller01:~ # neutron-db-manage revision --expand Running revision for neutron ... Generating /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py ... done OK controller01:~ # neutron-db-manage heads Running heads for neutron ... 6c9eb0469914 (expand) (head) c43a0ddb6a03 (contract) (head) OK ---snip--- I've been searching how that revision got into the database but I can't find any hint where that e4e236b0e1ff belongs to. To test the upgrade process I have a VM that was initially setup with Ocata, I'm able to investigate a snapshot from before the Pike upgrade, and I can't find a clue either, except for the neutron database. Although our current setup works just fine I'm not sure if this will be a show stopper for the Ussuri upgrade, although it wasn't for the previous upgrades. Anyway, how would I resolve this? I see these versions in the neutron.alembic_version table: +--------------+ | version_num | +--------------+ | 5c85685d616d | | e4e236b0e1ff | +--------------+ And 5c85685d616d seems to be a valid version from Newton, it seems. Could anyone provide some guidance on how to fix this before the next uprade? If possible I would like to avoid making manual changes in the database myself. Thanks and best regards, Eugen

2 3

[octavia] Xena PTG planning
by Gregory Thiemonge 30 Mar '21

30 Mar '21

Hi Octavia community! We have booked 2 sessions for Octavia during the Xena PTG: Tuesday April 20 - 13:00 - 17:00 UTC Wednesday April 21 - 13:00 - 17:00 UTC There's an etherpad for the PTG planning, please add your topics to the list: https://etherpad.opendev.org/p/xena-ptg-octavia Thanks, Greg

1 0

[tc][release] Networking-midonet current status and Wallaby release
by Slawek Kaplonski 30 Mar '21

30 Mar '21

Hi, We have opened release patch for networking-midonet [1] but our concern about that project is that its gate is completly broken since some time thus we don't really know if the project is still working and valid to be released. In Wallaby cycle Neutron for example finished transition to the engine facade, and patch to adjust that in networking-midonet is still opened [2] (and red as there were some unrelated issues with most of the jobs there). In the past we had discussion about networking-midonet project and it's status as the official Neutron stadium project. Then some new folks stepped in to maintain it but now it seems a bit like (again) it lacks of maintainers. I know that it is very late in the cycle now so my question to the TC and release teams is: should we release stable/wallaby with its current state, even if it's broken or should we maybe don't release it at all until its gate will be up and running? [1] https://review.opendev.org/c/openstack/releases/+/781713 [2] https://review.opendev.org/c/openstack/networking-midonet/+/770797 -- Slawek Kaplonski Principal Software Engineer Red Hat

6 19

Use of BladeCenter as a compute node(s)
by open infra 30 Mar '21

30 Mar '21

Hi, Done anyone deployed OpenStack compute nodes on BladeCenter? Do we need to attach each blade as a compute node? If we do so, aren't we restrict the maximum capacity of the VM to capacity of each blade? Regards, Danishka

1 0

[designate][ptg] Designate will have a PTG time for Xena
by Michael Johnson 29 Mar '21

29 Mar '21

Hi Designate community! I have reserved a timeslot to discuss Designate at the PTG. As discussed in the IRC channel, we will meet during the same time slot as we did for the last PTG. April 22th, 13:00-15:00 UTC I have created an etherpad to start collecting topics: https://etherpad.opendev.org/p/xena-ptg-designate I hope to see you there, Michael

1 0

[openstack-helm] Meeting cancelled March 31st
by Gage Hugo 29 Mar '21

29 Mar '21

Hey team, Since there are no agenda items [0] for the IRC meeting tomorrow, March 31st, the meeting is cancelled. Our next IRC meeting will be April 6th. Thanks [0] https://etherpad.opendev.org/p/openstack-helm-weekly-meeting

1 0

[puppet] Artificially inflated dependencies in metadata.json of all modules
by Thomas Goirand 29 Mar '21

29 Mar '21

Hi, Each time there's a new release, all of the modules are getting a new set of lower bounds for: - openstack/keystone - openstack/openstacklib - openstack/oslo I just did a quick check and this really doesn't make sense. For example, for going from 18.2.0 to 18.3.0, there's no breakage of the API that would require a bump of version in all modules. So, could we please stop this non-sense and restore some kind of sanity in our dependency management ? FYI, I'm expressing these in the packaged version of the puppet modules, and it increase complexity for no reasons. A lower bound of a dependency should be increased *only* when really mandatory (ie: a backward compatibility breakage, a change in the API, etc.). Your thoughts? Cheers, Thomas Goirand (zigo)

3 6

[tripleo] TripleO CI Summary: Unified Sprint 41
by Bhagyashri Shewale 29 Mar '21

29 Mar '21

Greetings, The TripleO CI team has just completed **Unified Sprint 41** (Feb 25 thru Mar 17 2021). The following is a summary of completed work during this sprint cycle: - Successfully deployed promoter server for all c8 and c7 releases using new code (next gen) promoter code <https://hackmd.io/team/rh-openstack-ci?nav=search&q=promoter> - Continue to adopt new changes and improvements in the promoter : - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32007 - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32069 - https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32058 - Reduce our upstream resource usage with zuul jobs including optimizations across the zuul layouts in all the tripleo-* repos - https://review.opendev.org/q/topic:tripleo-ci-reduce - Cirros image setup more resilience to infra setup to avoid failures: - Tempest conf image retry: https://review.opendev.org/c/osf/python-tempestconf/+/775195 - Usage of local cirros image: https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/778157 - Basically, the tag was wrong, and the copy and tagging was one single operation, now the copy command is copying every hour all the hashes available and tag is running separately now collecting the current-tripleo tag directly from the delorean api. - copy-quay script: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32478 - Migrations from CentOS-8 to Centos-8-stream: These are the reviews: https://hackmd.io/9Xve-rYpRaKbk5NMe7kukw - Modification of the private_apis of scenario manager: - Following apis have been made public by this commit as the tempest.scenario.manager interface is meant to be consumed by tempest plugins. - https://review.opendev.org/c/openstack/tempest/+/777089 - https://review.opendev.org/c/openstack/tempest/+/778697 - https://review.opendev.org/c/openstack/tempest/+/778698 - https://review.opendev.org/c/openstack/tempest/+/77735 <https://review.opendev.org/c/openstack/tempest/+/777351> - <https://review.opendev.org/c/openstack/tempest/+/777351>Design Upstream and downstream dependency pipeline dashboard: - Added unit tests for telegraf zuul v3 queue status script : https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/31948 - RPM dependency pipeline dashboard : https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32477/ - Created the design document for tripleo-repos <https://opendev.org/openstack/tripleo-repos> - https://hackmd.io/v2jCX9RwSeuP8EEFDHRa8g?view - https://review.opendev.org/c/openstack/tripleo-specs/+/772442 The planned work for the next sprint and leftover previous sprint work as following: - Deploy the promoter server for upstream promotions on vexxhost - Component / integration rdo pipelines for wallaby/ wallaby branching - Content provider jobs across all branches - Removing Rocky jobs (EOL) - elastic-recheck containerization - https://hackmd.io/dmxF-brbS-yg7tkFB_kxXQ - Openstack health for tripleo - https://hackmd.io/HQ5hyGAOSuG44Le2x6YzUw - Tripleo-repos spec and implementation - https://hackmd.io/v2jCX9RwSeuP8EEFDHRa8g?view - https://review.opendev.org/c/openstack/tripleo-specs/+/772442 - Tempest skiplist: Started to work on tempest-skiplist allowed list of tests (no patches yet) - Patrole stable release- Started to work on it The Ruck and Rover for this sprint are Chandan Kumar (chkumar) and Sorin (zbr). Please direct questions or queries to them regarding CI status or issues in #tripleo, ideally to whomever has the ‘|ruck’ suffix on their nick. Ruck/rover notes to be tracked in hackmd. Thanks, Bhagyashri Shewale

1 0

[neutron] Bug deputy report for week of March 22
by Lajos Katona 29 Mar '21

29 Mar '21

Hi, I was Neutron bug deputy for the previous week, please read a short summary of the reported bugs. Lajos - Critical - "test_add_remove_fixed_ip" faling in "grenade-dvr-multinode" CI job - https://bugs.launchpad.net/neutron/+bug/1920778 - grenade-dvr-multinode job is made non-voting - proposals to fix it: https://review.opendev.org/c/openstack/neutron/+/782677 & https://review.opendev.org/c/openstack/neutron/+/782690 - os.kill(SIGTERM) does not finish and timeouts - https://bugs.launchpad.net/neutron/+bug/1921154 - If I recall correctly the os.kill thing was reverted and decided to come back later to it: https://review.opendev.org/c/openstack/neutron/+/782972 - High Bugs - When OVS restart, some flows will be missing in the br-int - https://bugs.launchpad.net/neutron/+bug/1920700 - Merged / released - [OVN] Stale Logical Router Port entry fail to be removed by maintenance task - https://bugs.launchpad.net/neutron/+bug/1920968 - Merged / released - Rally test NeutronNetworks.create_and_update_subnets fails - https://bugs.launchpad.net/neutron/+bug/1920923 - In progress ( https://review.opendev.org/c/openstack/neutron/+/782587 ) - ovn: DVR on VLAN networks does not work - https://bugs.launchpad.net/neutron/+bug/1920976 - Merged / released - [QoS min bw] repeated ERROR log: Unable to save resource provider ... because: re-parenting a provider is not currently allowed - https://bugs.launchpad.net/neutron/+bug/1921150 - In progress: https://review.opendev.org/c/openstack/neutron/+/782553 - Medium Bugs - "rpc_response_max_timeout" configuration variable not present in metadata-agent - https://bugs.launchpad.net/neutron/+bug/1920842 - Merge / releases - neutron dvr should lower proxy_delay when using proxy_arp - https://bugs.launchpad.net/neutron/+bug/1920975 - In progress ( https://review.opendev.org/c/openstack/neutron/+/782570 ) - ml2/ovn should not be calling neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api - https://bugs.launchpad.net/neutron/+bug/1921491 - Unassigned - VPNaaS strongSwan driver does not reload secrets - https://bugs.launchpad.net/neutron/+bug/1921514 - In progress in neutron-vpnaas: https://review.opendev.org/c/openstack/neutron-vpnaas/+/783331 - Low - RFE/Whislist - [RFE] Allow explicit management of default routes - https://bugs.launchpad.net/neutron/+bug/1921126 - approved - [RFE] Enhancement to Neutron BGPaaS to directly support Neutron Routers & bgp-peering from such routers over internal & external Neutron Networks - https://bugs.launchpad.net/neutron/+bug/1921461 - Approved - - Invalid - neutron-server ovsdbapp timeout exceptions after intermittent connectivity issues - https://bugs.launchpad.net/neutron/+bug/1921085 - Designate PTR record creation results in in-addr.arpa. zone owned by invalid project ID - https://bugs.launchpad.net/neutron/+bug/1921414 - 'Table 'ovn_revision_numbers' is already defined for this MetaData instance - https://bugs.launchpad.net/neutron/+bug/1921577

1 0

Ospurge or "project purge" - What's the right approach to cleanup projects prior to deletion
by Christian Rohmann 29 Mar '21

29 Mar '21

Hello Openstack-Discuss, I was wondering what the right approach to clean up all resources of a project was in order for it to be deleted? 1) There is ospurge (https://opendev.org/x/ospurge) which works, but seems kind of abandoned looking at the last commits and pending issues (https://bugs.launchpad.net/ospurge) Will there still be any further development of this tool? 2) Then there is the project purge command (https://docs.openstack.org/python-openstackclient/victoria/cli/command-obje…) introduced with the Pike release (https://docs.openstack.org/releasenotes/python-openstackclient/pike.html#re… // https://bugs.launchpad.net/python-openstackclient/+bug/1584596 // https://review.opendev.org/c/openstack/python-openstackclient/+/367673/) While it seems only natural to have the cleanup / purge functionality built into the openstack client, I was wondering .. . a) if all potentially existing resources types are covered and this can fully replace ospurge to allow any project to be cleaned up and deleted? What about i.e. Networks, Containers or objects? b) Also ospurge seems to still have some extended options to target or exclude certain resources by using the following options: > [--resource > {Backups,Snapshots,Volumes,Zones,Images,Stacks,FloatingIPs,RouterInterfaces,Routers,Ports,Networks,SecurityGroups,Servers,LoadBalancers,Receivers,Policies,Clusters,Profiles,Objects,Containers}] > [--exclude-resource > {Backups,Snapshots,Volumes,Zones,Images,Stacks,FloatingIPs,RouterInterfaces,Routers,Ports,Networks,SecurityGroups,Servers,LoadBalancers,Receivers,Policies,Clusters,Profiles,Objects,Containers}] those options are not (yet) available for the "project purge" command of the openstack client. With regards, Christian

5 13

[all][tc] Xena PTG Planning
by Ghanshyam Mann 29 Mar '21

29 Mar '21

Hello Everyone, As you already know that the Xena cycle virtual PTG will be held between 19th - 23rd April[1]. To plan the Technical Committee PTG planning, please do the following: 1. Fill the doodle poll as per your availability. Please fill it soon as we need to book the slot by 25th March(which is tomorrow). - https://doodle.com/poll/2zy8hex4r6wvidqk?utm_source=poll&utm_medium=link 2. Add the topics you would like to discuss to the below etherpad. - https://etherpad.opendev.org/p/tc-xena-ptg NOTE: this is not limited to TC members only; I would like all community members to fill the doodle poll and, add the topics you would like or want TC members to discuss in PTG. -gmann

2 3

[tc] New Chair
by Mohammed Naser 29 Mar '21

29 Mar '21

Hi everyone, I'd like to introduce/congratulate/welcome Ghanshyam as the new chair of our technical committee. I'm happy to see them step up to this role and I think they're an excellent candidate from their hard work within the community. Thanks for having me as chair for the past few years, I'm still around and continue to be on the technical committee and thanks for reading my weekly updates :) Regards, Mohammed -- Mohammed Naser VEXXHOST, Inc.

4 3

[Containers SIG][tc]Is any one interested on Containers SIG? Or should we retire Containers SIG?
by Rico Lin 29 Mar '21

29 Mar '21

Dear all, I just propose a patch to openstack/governance-sigs to officially retire containers SIG [1]. In past months, there's no activity in the SIG, and Jean-Philippe didn't get time for the SIG chair role too. IMO, The goal from Containers SIG definitely provide great value to Openstack community because we didn't have much work on checking whether we reach great container support or not. Hance don't really wish it got archived and retired. Still if no one interested enough and have time to join and chair the SIG, we have no option but to move it to retired state ("archived"). So if you're interested in chair the SIG, please raise your hand now. For sure SIGs might be able to bring back to live after we retired it. But I guess will be even better if we got someone take it over and work on it. Again if no one volunteer to chair it, the retire patch [1] will be the way to go. [1] https://review.opendev.org/c/openstack/governance-sigs/+/783001 *Rico Lin* OIF Board director, OpenSack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack *Email: ricolin(a)ricolky.com <ricolin(a)ricolky.com>* *Phone: +886-963-612-021*

2 2

[kolla-ansible] Inventory group vars
by Alexandros Soumplis 27 Mar '21

27 Mar '21

Hi all, I am trying to define a couple of variables in the inventory file for a specific group and I do something like the following in the inventory: (...) [compute] comp[1:5] [compute:vars] neutron_external_interface=bond0.100, bond0.101 neutron_bridge_name=br-ex,br-ex2 (...) This is a valid ansible setup, however with kolla-ansible these variables are totally ignored. Any suggestion ? a.

4 6

[Keystone][Oslo] Policy problems
by meberhardt＠unl.edu.ar 26 Mar '21

26 Mar '21

Hi, my installation complains about deprecated policies and throw errors when I try to run spacific commands in cli (list projects or list users, for example). "You are not authorized to perform the requested action: identity:list_users." I tried to fix this by upgrading the keystone policies using oslopolicy-policy-generator and oslopolicy-policy-upgrade. Just found two places where I have those keystone policy files in my sistem: /etc/openstack_dashboard/keystone_policy.json and cd /lib/python3.6/site-packages/openstack_auth/tests/conf/keystone_policy.json I regenerated & upgraded it but Keystone still complains about the old polices. ¿Where are it placed? ¿How I shoud fix it? OS: CentOS 8 Openstack version: Ussuri Manual installation Thanks, Matias /* --------------------------------------------------------------- */ /* Matías A. Eberhardt */ /* */ /* Centro de Telemática */ /* Secretaría General */ /* UNIVERSIDAD NACIONAL DEL LITORAL */ /* Pje. Martínez 2652 - S3002AAB Santa Fe - Argentina */ /* tel +54(342)455-4245 - FAX +54(342)457-1240 */ /* --------------------------------------------------------------- */

2 1

[PTLs][All] vPTG April 2021 Team Signup
by Kendall Nelson 26 Mar '21

26 Mar '21

Greetings! As you hopefully already know, our next PTG will be virtual again, and held from Monday, April 19 to Friday, April 23. We will have the same schedule set up available as last time with three windows of time spread across the day to cover all timezones with breaks in between. *To signup your team, you must complete **BOTH** the survey[1] AND reserve time in the ethercalc[2] by March 25 at 7:00 UTC.* We ask that the PTL/SIG Chair/Team lead sign up for time to have their discussions in with 4 rules/guidelines. 1. Cross project discussions (like SIGs or support project teams) should be scheduled towards the start of the week so that any discussions that might shape those of other teams happen first. 2. No team should sign up for more than 4 hours per UTC day to help keep participants actively engaged. 3. No team should sign up for more than 16 hours across all time slots to avoid burning out our contributors and to enable participation in multiple teams discussions. Again, you need to fill out BOTH the ethercalc AND the survey to complete your team's sign up. If you have any issues with signing up your team, due to conflict or otherwise, please let me know! While we are trying to empower you to make your own decisions as to when you meet and for how long (after all, you know your needs and teams timezones better than we do), we are here to help! Once your team is signed up, please register! And remind your team to register! Registration is free, but since it will be how we contact you with passwords, event details, etc. it is still important! Continue to check back for updates at openstack.org/ptg. -the Kendalls (diablo_rojo & wendallkaters) [1] Team Survey: https://openinfrafoundation.formstack.com/forms/april2021_vptg_survey [2] Ethercalc Signup: https://ethercalc.net/oz7q0gds9zfi [3] PTG Registration: https://april2021-ptg.eventbrite.com

4 7

Cannot specify host in availability zone
by Mauricio Tavares 26 Mar '21

26 Mar '21

Easy peasy question: According to https://docs.openstack.org/nova/rocky/admin/availability-zones.html, I can specify the host I want to use by following the format --availability-zone ZONE:HOST So, let's get the hostnames for the compute nodes. [raub@test-hn ~(keystone_admin)]$ openstack compute service list --service nova-compute +----+--------------+-----------------------+------+---------+-------+----------------------------+ | ID | Binary | Host | Zone | Status | State | Updated At | +----+--------------+-----------------------+------+---------+-------+----------------------------+ | 11 | nova-compute | compute02.example.com | nova | enabled | up | 2021-02-03T17:25:53.000000 | | 12 | nova-compute | compute03.example.com | nova | enabled | up | 2021-02-03T17:25:53.000000 | | 13 | nova-compute | compute01.example.com | nova | enabled | up | 2021-02-03T17:25:52.000000 | +----+--------------+-----------------------+------+---------+-------+----------------------------+ [raub@test-hn ~(keystone_admin)]$ I want to use compute01.example.com, which I can resolve to 10.1.1.11. But, when I try to create server with (running as admin): openstack server create \ --image default_centos_8 \ --flavor m1.small \ --key-name raubkey \ --availability-zone nova:compute01.example.com \ --nic net-id=LONG_BORING-ID \ raub-netest I get the error message (from openstack server show raub-netest|grep fault): 'message': 'No valid host was found. No such host - host: compute01.example.com node: None ' What am I doing wrong here?

3 4

[cinder] review priorities for week of 29 march
by Brian Rosmaita 26 Mar '21

26 Mar '21

We're keeping track of release-critical patches here: https://etherpad.opendev.org/p/cinder-wallaby-release-critical-bug-nominati… The stable/wallaby branch was cut yesterday when RC-1 was released. So these must be reviewed and merged to master and then proposed as backports to stable/wallaby and reviewed and approved again. The week of 29 March is a short week for people in various parts of the world, and Monday the week after (which is final RC week) is also a holiday for some people. So even though it seems like there's a lot of time to prepare RC-2, there really isn't. Please schedule some time to do reviews early in the week. cheers, brian

1 0

[cinder] critical os-brick fix needs immediate review
by Brian Rosmaita 26 Mar '21

26 Mar '21

Gorka has a patch up to address a possible data loss situation: https://review.opendev.org/c/openstack/os-brick/+/782992 I spoke with the release team today, and if we can get the patch reviewed and merged to master and then backported to stable/wallaby by MONDAY (29 March), we can apply for a Requirements Freeze Exception so that os-brick with this fix will be included in the OpenStack coordinated wallaby release. (It's a data loss issue, of course we want the fix in the wallaby release.) So please review at your earliest convenience. thanks, brian

1 1

[manila] RC1 is available, No IRC meeting on 1st April 2021
by Goutham Pacha Ravi 26 Mar '21

26 Mar '21

Hello Zorillas, RC1 builds have now shipped for both manila and manila-ui, thank you all for your hard work on getting these out on time. It's now time to test these builds and learn of any release blocking bugs and fix them. Some down time is also in order between now and the PTG! In order to let folks to plan for that, I'd like to skip the IRC meeting on 1st April. In the meantime, please feel free to post your issues to #openstack-manila and to this email list. Thanks, Goutham

1 0

[TripleO] next irc meeting Tuesday Mar 30 @ 1400 UTC in #tripleo
by Marios Andreou 26 Mar '21

26 Mar '21

Reminder that the next TripleO irc meeting is: ** Tuesday 30 March at 1400 UTC in #tripleo ** ** https://wiki.openstack.org/wiki/Meetings/TripleO ** ** https://etherpad.opendev.org/p/tripleo-meeting-items ** Please add anything you want to highlight at https://etherpad.opendev.org/p/tripleo-meeting-items This can be recently completed things, ongoing review requests, blocking issues, or anything else tripleo you want to share. Our last meeting was on Mar 16 - you can find the logs there http://eavesdrop.openstack.org/meetings/tripleo/2021/tripleo.2021-03-16-14.… Hope you can make it on Tuesday, regards, marios

1 0

[kolla] Train Centos7 -> Centos8 upgrade fails on masakari
by Braden, Albert 26 Mar '21

26 Mar '21

I've created a heat stack and installed Openstack Train to test the Centos7->8 upgrade following the document here: https://docs.openstack.org/kolla-ansible/train/user/centos8.html#migrating-… Everything seems to work fine until I try to deploy the first replacement controller into the cluster. I upgrade RMQ, ES and Kibana, then follow the "remove existing controller" process to remove control0, create a Centos 8 VM, bootstrap the cluster, pull containers to the new control0, and everything is still working. Then I type the last command "kolla-ansible -i multinode deploy --limit control" The RMQ install works and I see all 3 nodes up in the RMQ admin, but it takes a long time to complete "TASK [service-ks-register : masakari | Creating users] " and then hangs on "TASK [service-ks-register : masakari | Creating roles]". At this time the new control0 becomes unreachable and drops out of the RMQ cluster. I can still ping it but console hangs along with new and existing ssh sessions. It appears that the CPU may be maxed out and not allowing interrupts. Eventually I see error "fatal: [control0]: FAILED! => {"msg": "Timeout (12s) waiting for privilege escalation prompt: "}" RMQ seems fine on the 2 old controllers; they just don't see the new control0 active: (rabbitmq)[root@chrnc-void-testupgrade-control-2 /]# rabbitmqctl cluster_status Cluster status of node rabbit@chrnc-void-testupgrade-control-2 ... [{nodes,[{disc,['rabbit@chrnc-void-testupgrade-control-0', 'rabbit@chrnc-void-testupgrade-control-0-replace', 'rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}]}, {running_nodes,['rabbit@chrnc-void-testupgrade-control-1', 'rabbit@chrnc-void-testupgrade-control-2']}, {cluster_name,<<"rabbit(a)chrnc-void-testupgrade-control-0.dev.chtrse.com">>}, {partitions,[]}, {alarms,[{'rabbit@chrnc-void-testupgrade-control-1',[]}, {'rabbit@chrnc-void-testupgrade-control-2',[]}]}] After this the HAProxy IP is pingable but openstack commands are failing: (openstack) [root@chrnc-void-testupgrade-build openstack]# osl Failed to discover available identity versions when contacting http://172.16.0.100:35357/v3. Attempting to parse version from URL. Gateway Timeout (HTTP 504) After about an hour my open ssh session on the new control0 responded and confirmed that the CPU is maxed out: [root@chrnc-void-testupgrade-control-0-replace /]# uptime 17:41:55 up 1:55, 0 users, load average: 157.87, 299.75, 388.69 I built new heat stacks and tried it a few times, and it consistently fails on masakari. Do I need to change something in my masakari config before upgrading Train from Centos 7 to Centos 8? I apologize for the nonsense below. I have not been able to stop it from being attached to my external emails. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

4 5

[release] Release countdown for week R-2 Mar 29 - Apr 02
by Herve Beraud 26 Mar '21

26 Mar '21

Development Focus ----------------- At this point we should have release candidates (RC1 or recent intermediary release) for all the Wallaby deliverables. Teams should be working on any release-critical bugs that would require another RC or intermediary release before the final release. Actions ------- Early in the week, the release team will be proposing stable/wallaby branch creation for all deliverables that have not branched yet, using the latest available Wallaby release as the branch point. If your team is ready to go for creating that branch, please let us know by leaving a +1 on these patches. If you would like to wait for another release before branching, you can -1 the patch and update it later in the week with the new release you would like to use. By the end of the week the release team will merge those patches though, unless an exception is granted. Once stable/wallaby branches are created, if a release-critical bug is detected, you will need to fix the issue in the master branch first, then backport the fix to the stable/wallaby branch before releasing out of the stable/wallaby branch. After all of the cycle-with-rc projects have branched we will branch devstack, grenade, and the requirements repos. This will effectively open them up for Xena development, though the focus should still be on finishing up Wallaby until the final release. For projects with translations, watch for any translation patches coming through and merge them quickly. A new release should be produced so that translations are included in the final Wallaby release. Finally, now is a good time to finalize release notes. In particular, consider adding any relevant "prelude" content. Release notes are targetted for the downstream consumers of your project, so it would be great to include any useful information for those that are going to pick up and use or deploy the Wallaby version of your project. Upcoming Deadlines & Dates -------------------------- Final RC deadline: 08 April, 2021 (R-1 week) Final Wallaby release: 14 April, 2021 Xena virtual PTG: 19 - 23 April, 2021 Thank you for your attention -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

1 0

[nova][placement] Wallaby release
by Balazs Gibizer 26 Mar '21

26 Mar '21

Hi, We are getting close to the Wallaby release. So I create a tracking etherpad[1] with the schedule and TODOs. One thing that I want to highlight is that we will hit Feature Freeze on 11th of March. As the timeframe between FF and RC1 is short I'm not plannig with FFEs. Patches that are approved before 11 March EOB can be rechecked or rebased if needed and then re-approved. If you have a patch that is really close but not approved before the deadline, and you think there are two cores that willing to review it before RC1, then please send a mail to the ML with [nova][FFE] subject prefix not later than 16th of March EOB. Cheers, gibi [1] https://etherpad.opendev.org/p/nova-wallaby-rc-potential

2 3

[glance] [ops] Glance deployed on 2 nodes with rbd. Do I need a shared filesystem for the reserved stores ?
by Massimo Sgaravatto 26 Mar '21

26 Mar '21

We are using OpenStack Train but we plan to migrate soon to something newer The glance-api service is deployed on 2 nodes. We used to have 2 backends for glance (file and rbd) but now we want to use only ceph. So I guess the configuration for the stores can be simply something like this [*] Reading: https://docs.openstack.org/glance/train/admin/multistores.html is not too clear to me if I have to configure somewhere filesystem_store_datadir. In particular do I have to configure the [os_glance_tasks_store] and [os_glance_staging_store] reserved stores ? If so, do I still need shared (between the 2 glance-api nodes )filesystems for them ? Thanks, Massimo [*] [DEFAULT] enabled_backends = rbd:rbd [glance_store] default_backend = rbd [rbd] store_description = Ceph in replica 3 rbd_store_pool = glance-cloudtest rbd_store_user = glance-cloudtest rbd_store_ceph_conf = /etc/ceph/ceph.conf rbd_store_chunk_size = 8

2 3

[cyborg][ptg] Xena PTG
by Wang, Xin-ran 25 Mar '21

25 Mar '21

Hi all, we signed up for the following slots [1] at Cactus room: * Tue Apr. 20th 6:00- 8:00 UTC (2 slots) * Wed Apr. 21th 6:00- 8:00 UTC (2 slots) * Thu Apr. 22th 6:00- 8:00 UTC (2 slots) * Fri Apr. 23th 6:00- 8:00 UTC (2 slots) If you want add some topic to discuss, please feel free to add it to the etherpad [2]. [1] https://ethercalc.net/oz7q0gds9zfi [2] https://etherpad.opendev.org/p/cyborg-xena-ptg Thanks, Xin-Ran

1 0

[all][elections][tc] TC Vacancy Special Election Nominations Kickoff
by Jeremy Stanley 25 Mar '21

25 Mar '21

Nominations for one vacant OpenStack TC (Technical Committee) position are now open and will remain open until Apr 08, 2021 23:45 UTC. All nominations must be submitted as a text file to the openstack/election repository as explained at https://governance.openstack.org/election/#how-to-submit-a-candidacy Please make sure to follow the candidacy file naming convention: candidates/xena/<Project_Name or "TC">/<email_address> (for example, "candidates/xena/TC/stacker(a)example.org"). The name of the file should match an email address for your current OpenStack Foundation Individual Membership. Take this opportunity to ensure that your OSF member profile contains current information: https://www.openstack.org/profile/ Any OpenStack Foundation Individual Member can propose their candidacy for the vacant seat on the Technical Committee. This TC vacancy special election will be held from Apr 8, 2021 23:45 UTC through to Apr 15, 2021 23:45 UTC. The electorate for the TC election are the OpenStack Foundation Individual Members who have a code contribution to one of the official teams over the Victoria to Wallaby timeframe, Apr 24, 2020 00:00 UTC - Mar 08, 2021 00:00 UTC, as well as any Extra ATCs who are acknowledged by the TC. Note that the contribution qualifying period for this special election is being kept the same as what would have been used for the original TC election. The four already elected TC members for this term are listed as candidates in the special election, but will not appear on any resulting poll as they have already been officially elected. Only new candidates in addition to the four elected TC members for this term will appear on a subsequent poll for the TC vacancy special election. Please find below the timeline: nomination starts @ Mar 25, 2021 23:45 UTC nomination ends @ Apr 08, 2021 23:45 UTC elections start @ Apr 08, 2021 23:45 UTC elections end @ Apr 15, 2021 23:45 UTC Shortly after election officials approve candidates, they will be listed on the https://governance.openstack.org/election/ page. The electorate is requested to confirm their preferred email addresses in Gerrit prior to 2021-04-01 00:00:00+00:00, so that the emailed ballots are sent to the correct email address. This email address should match one which was provided in your foundation member profile as well. Gerrit account information and OSF member profiles can be updated at https://review.openstack.org/#/settings/contact and https://www.openstack.org/profile/ accordingly. If you have any questions please be sure to either ask them on the mailing list or to the elections officials: https://governance.openstack.org/election/#election-officials -- Jeremy Stanley on behalf of the OpenStack Technical Elections Officials

1 0

[cinder] need core reviewers for small driver fix
by Belogrudov, Vladislav 25 Mar '21

25 Mar '21

Dear core reviewers, I would like to ask for help in reviewing a small bug fix in Cinder driver for Dell EMC PowerStore. It's just 2 chars, the driver wrongly used "eq" operator in PostgREST request instead of "cs" for finding a value in an array. The fix is under review [1]. Official API guide for PowerStore [2] says that ip_pool_address endpoint returns address instances with array of purposes, where the driver searches for iSCSI purpose. If there are more purposes configured for an iSCSI target address, the driver fails to operate. It would be great if the fix could go to Wallaby. Best regards, Vladislav Belogrudov [1] https://review.opendev.org/c/openstack/cinder/+/782087 [2] https://downloads.dell.com/manuals/common/pwrstr-apig_en-us.pdf

2 1

[infra][stable] Request for Designate stable core status
by Michael Johnson 25 Mar '21

25 Mar '21

I would like to request stable core status for the Designate repositories. As PTL I am currently unable to approve even the release bot patches for some Designate stable branches[1]. I have been a stable reviewer for Octavia for years and if there are concerns about my stable policy track record I hope that my work on Octavia would highlight my respect for the stable release policy. Thanks, Michael [1] https://review.opendev.org/c/openstack/designate-dashboard/+/783056

2 1

[tripleo][ci] tripleo gate is hung
by Wesley Hayutin 25 Mar '21

25 Mar '21

Greetings, https://zuul.openstack.org/status top of the gate patch is hung 774500,1 Infra is looking into it. See #openstack-infra for details Thanks fungi!!

2 2

[qa][ptg] Xena PTG
by Martin Kopec 25 Mar '21

25 Mar '21

Hello, we signed up for the following slots [1] at Cactus room: * Mon Apr. 19th 13 - 15 UTC (2 slots) * Tue Apr. 20th 13 - 14 UTC (1 slot) * Wed Apr. 21th 13 - 14 UTC (1 slot) The PTG etherpad [2] with the topics we have so far. If you have something else you wanna discuss, feel free to add it to the agenda [2]. [1] https://etherpad.opendev.org/p/qa-xena-ptg [2[ https://etherpad.opendev.org/p/qa-xena-ptg See you at PTG, -- *Martin Kopec*

1 0

[policy pop-up] Xena PTG planning
by Ghanshyam Mann 25 Mar '21

25 Mar '21

Hello Everyone, We have booked 2 hrs slot for the policy popup team in the Xena virtual PTG. The time for the slot is 15.00-17.00 UTC Monday, April 19 (2 hours; Essex Room). I have created the etherpad to collect the discussion point. Please add the topic you would like to discuss - https://etherpad.opendev.org/p/policy-popup-xena-ptg We request more and more projects to participate in this and discuss the various queries or plan to switch to the new policy in a single place instead of duplicating the discussion in each project slot. -gmann & lbragstad

1 0

[ops][victoria] DNS for Tenants
by DHilsbos＠performair.com 25 Mar '21

25 Mar '21

All; How do we provide DNS services to tenant networks? We're running Victoria, and have Designate installed and configured, but it doesn't seem to have a presence on the tenant networks. I assume if the instance can route to an IP on the network node then the installed Bind can be used. I'd rather not have to deal with this, however. I would like the DNS to operate in a relay fashion to our previously configured DNS servers, though I assume that standard Bind configuration for relay should take care of this. Thank you, Dominic L. Hilsbos, MBA Director - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

2 3

[TripleO] Xena PTG is coming 19 April
by Marios Andreou 25 Mar '21

25 Mar '21

Hello tripleo friends o/ It's PTG time again \o/ and I have started reaching out to/harassing folks this week about topics. I want to send a *HUGE* thanks to all the usual suspects who have already proposed some sessions. Like last time let's collect all the topics into the topic etherpad at [1] and then around Monday 5th April or so we can start to work out the schedule (2 weeks before PTG). I have reserved 1300-1700 UTC on Monday/Tue/Wed/Thu as noted there [2] - we may not need all of those days it depends how many sessions we will have proposed. So over the next few days please think about the Wallaby or Xena tripleo related things you'd like to discuss/provide updates on/socialise with the wider TripleO team. Then go add them into [1] using the template there (Topic/Description/Etherpad Link/Proposer). BUT before you do any of that make sure you go and register for PTG at [3]. I hope we get a great turn out like last time! regards, marios [1] https://etherpad.opendev.org/p/tripleo-xena-topics [2] http://lists.openstack.org/pipermail/openstack-discuss/2021-March/020953.ht… [3] https://april2021-ptg.eventbrite.com/

1 0

[telemetry] OpenStack Telemetry Xena PTG planning
by Matthias Runge 25 Mar '21

25 Mar '21

Hi there, The next virtual PTG is getting near very quickly. If you haven't registered yet, please do so as soon as possible[1]. We have signed up for slots on Tuesday and Wednesday April 20 and 21 for the slots between 13 and 16 UTC. If there is anything you would like to talk about or something you'd like to address/to work on, please feel free to add it to the etherpad[2]. Thank you and see you there! Matthias [1] https://april2021-ptg.eventbrite.com/ [2] https://etherpad.opendev.org/p/telemetry-xena-ptg

1 0

[oslo.db][Zaqar] test_models_sync failed because oslo.db raise "AttributeError: can't set attribute"
by hao wang 25 Mar '21

25 Mar '21

Hi, There is an issue about test_models_sync test when openstack-tox-lower-constraints job running that I can't figure out why it's here, could anyone help me about this? zaqar.tests.unit.storage.sqlalchemy_migration.test_migrations.TestMigrationsMySQL.test_models_sync [0.306333s] ... FAILED Captured traceback: ~~~~~~~~~~~~~~~~~~~ Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/oslo_db/sqlalchemy/test_base.py", line 178, in setUp self.useFixture( File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testtools/testcase.py", line 760, in useFixture reraise(*exc_info) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testtools/_compat3x.py", line 16, in reraise raise exc_obj.with_traceback(exc_tb) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testtools/testcase.py", line 735, in useFixture fixture.setUp() File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/oslo_db/sqlalchemy/test_base.py", line 65, in setUp testresources.setUpResources( File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testresources/__init__.py", line 786, in setUpResources setattr(test, resource[0], resource[1].getResource(result)) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testresources/__init__.py", line 522, in getResource self._setResource(self._make_all(result)) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testresources/__init__.py", line 551, in _make_all dependency_resources[name] = resource.getResource() File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testresources/__init__.py", line 522, in getResource self._setResource(self._make_all(result)) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/testresources/__init__.py", line 552, in _make_all resource = self.make(dependency_resources) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/oslo_db/sqlalchemy/provision.py", line 133, in make url = backend.provisioned_database_url(db_token) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/oslo_db/sqlalchemy/provision.py", line 337, in provisioned_database_url return self.impl.provisioned_database_url(self.url, ident) File "/home/zuul/src/opendev.org/openstack/zaqar/.tox/lower-constraints/lib/python3.8/site-packages/oslo_db/sqlalchemy/provision.py", line 498, in provisioned_database_url url.database = ident AttributeError: can't set attribute There is the test log: https://zuul.opendev.org/t/openstack/build/57eb103379b144abaa2517d41ef67610…

3 2

[kolla-ansible][Neutron]Limiting bandwidth per Project/Domain/Tenant?
by Adam Tomas 25 Mar '21

25 Mar '21

Hi, Is there any way to limit bandwidth per project, domain or tenant? According to the neutron documentation limit can be set per port (router,FIP etc.) but what if I need to limit whole project (which can have multiple routers, FIPs, etc.) - incoming and outgoing traffic? Best regards, Adam

2 1

[neutron][stable] neutron-lib stable branches core reviewers
by Slawek Kaplonski 25 Mar '21

25 Mar '21

Hi, I just noticed that neutron-lib project has got own group "neutron-lib-stable-maint" which has +2 powers in neutron-lib stable branches [1]. As I see now in gerrit that group don't have any members. Would it be maybe possible to remove that group and add "neutron-stable-maint" to the neutron-lib stable branches instead? If yes, should I simply propose patch to change [1] or is there any other way which I should do it? [1] https://github.com/openstack/project-config/blob/master/gerrit/acls/opensta… -- Slawek Kaplonski Principal Software Engineer Red Hat

4 5

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 25 Mar '21

25 Mar '21

Hi Dmitriy, thanks for your answer! I tried inserting the options into user_variables.yml and running the installation playbooks again but glance still shows 'Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token.' and the snapshot keeps disappearing. In addition, I'm seeing the following swift error: Mar 25 11:19:56 infra1-swift-proxy-container-27169fa7 proxy-server[90]: Pipeline was modified. New pipeline is "catch_errors gatekeeper healthcheck proxy-logging cache listing_formats container_sync bulk tempurl ratelimit authtoken keystoneauth staticweb copy container-quotas account-quotas slo dlo versioned_writes proxy-logging proxy-server". Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: Starting Keystone auth_token middleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "use_stderr" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "bind_ip" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "bind_port" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "workers" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "user" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "log_name" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "auth_url" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "project_domain_id" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "user_domain_id" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "project_name" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "username" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "password" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "revocation_cache_time" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: STDERR: The option "__name__" is not known to keystonemiddleware Mar 25 11:19:57 infra1-swift-proxy-container-27169fa7 proxy-server[90]: AuthToken midd Kind regards Oliver > Oh, I have a guess what this might actually be. During snapshot upload process > user token that is used for the upload might get expired. If that's the case, > following changes in user_variables might help to resolve the issue: > > glance_glance_api_conf_overrides: > keystone_authtoken: > service_token_roles_required: True > service_token_roles: service >

1 0

Re: [glance][openstack-ansible] Snapshots disappear during saving
by Oliver Wenz 25 Mar '21

25 Mar '21

I manually set the fallocate_reserve values to 1% on the swift host and the services don't show errors anymore (I also tried with higher values than 1%). However, when trying to take the snapshot it still disappears and I still get the following for the swift-proxy service: Mar 25 08:55:53 infra1-swift-proxy-container-27169fa7 proxy-server[84]: Client disconnected without sending last chunk (txn: tx5590a6d463f645caaaf67-00605c4fe4) (client_ip: 192.168.110. 106) 192.168.110.106 is the IP of eth1 on my glance container so maybe that indicates that this is a glance problem after all? Because of the following log from the glance-api service: Mar 25 08:54:59 infra1-glance-container-99614ac2 glance-wsgi-api[6177]: 2021-03-25 08:54:59.046 6177 INFO glance.api.v2.image_data [req-42a0c48b-895c-4776-9e9c-586eb596b540 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Unable to create trust: no such option collect_timing in group [keystone_authtoken] Use the existing user token. Mar 25 08:55:54 infra1-glance-container-99614ac2 uwsgi[6177]: Thu Mar 25 08:55:54 2021 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2/images/16f50adc-ebec-4812-96bd-2bbf6d2014b5/file (ip 192.168.110.214) !!! I checked the logs on the compute host (192.168.110.214) where the instance I'm trying to take a snapshot of is running and I found this in the logs: Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.050 2252 INFO nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - - -] image c27e2055-8c3c-49d7-9c12-44999b1e7e0f at (/var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478): checking Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.052 2252 INFO nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - - -] image e73512b8-7099-4691-aa48-966aa35b59ff at (/var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3): checking Mar 25 08:55:15 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:15.759 2252 INFO nova.virt.libvirt.imagecache [req-24323bc9-65bc-47b6-8ef1-81fc88b2b267 - - - - -] Active base files: /var/lib/nova/instances/_base/4102ded7765e7306705acece9b2b1c4e88087478 /var/lib/nova/instances/_base/573a84fb0420a038702c616b848169b07e2bb7f3 Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.102 2252 INFO nova.compute.manager [req-42a0c48b-895c-4776-9e9c-586eb596b540 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] [instance: 46d45c54-eba1-4624-8e5a-19dc157484ae] Successfully reverted task state from image_uploading on failure for instance. Mar 25 08:55:57 bc1bl14 nova-compute[2252]: 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server [req-42a0c48b-895c-4776-9e9c-586eb596b540 956806468e9f43dbaad1807a5208de52 ebe0fe5f3893495e82598c07716f5d45 - default default] Exception during message handling: glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2478, in snapshot 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server metadata['location'] = root_disk.direct_snapshot( 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/imagebackend.py", line 452, in direct_snapshot 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise NotImplementedError(_('direct_snapshot() is not implemented')) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server NotImplementedError: direct_snapshot() is not implemented 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 309, in dispatch 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_messaging/rpc/dispatcher.py", line 229, in _do_dispatch 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 76, in wrapped 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server _emit_exception_notification( 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise value 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/exception_wrapper.py", line 69, in wrapped 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return f(self, context, *args, **kw) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 188, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server LOG.warning("Failed to revert task state for instance. " 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise value 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 159, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/utils.py", line 1434, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 216, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server compute_utils.add_instance_fault_from_exc(context, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise value 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 205, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return function(self, context, *args, **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 236, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server compute_utils.delete_image( 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self.force_reraise() 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise value 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 232, in decorated_function 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return function(self, context, image_id, instance, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3908, in snapshot_instance 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self._snapshot_instance(context, image_id, instance, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/compute/manager.py", line 3941, in _snapshot_instance 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self.driver.snapshot(context, instance, image_id, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/virt/libvirt/driver.py", line 2549, in snapshot 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server self._image_api.update(context, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1247, in update 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return session.update(context, image_id, image_info, data=data, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 696, in update 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server _reraise_translated_image_exception(image_id) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 1035, in _reraise_translated_image_exception 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise new_exc.with_traceback(exc_trace) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 694, in update 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server image = self._update_v2(context, sent_service_image_meta, data) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 713, in _update_v2 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server image = self._upload_data(context, image_id, data) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 589, in _upload_data 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server utils.tpool_execute(self._client.call, 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/utils.py", line 694, in tpool_execute 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server eventlet.tpool.execute(func, *args, **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 129, in execute 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server six.reraise(c, e, tb) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/six.py", line 703, in reraise 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise value 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/eventlet/tpool.py", line 83, in tworker 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server rv = meth(*args, **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/nova/image/glance.py", line 192, in call 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server result = getattr(controller, method)(*args, **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/utils.py", line 600, in inner 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return RequestIdProxy(wrapped(*args, **kwargs)) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/v2/images.py", line 289, in upload 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server resp, body = self.http_client.put(url, headers=hdrs, data=body) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/keystoneauth1/adapter.py", line 404, in put 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return self.request(url, 'PUT', **kwargs) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 380, in request 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server return self._handle_response(resp) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server File "/openstack/venvs/nova-22.1.0/lib/python3.8/site-packages/glanceclient/common/http.py", line 120, in _handle_response 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server raise exc.from_response(resp, resp.content) 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server glanceclient.exc.HTTPInternalServerError: HTTP 500 Internal Server Error: The server has either erred or is incapable of performing the requested operation. 2021-03-25 08:55:57.106 2252 ERROR oslo_messaging.rpc.server Any hints where the error might be coming from are much appreciated! Kind regards, Oliver

2 1

[neutron] Drivers meeting agenda for 26.03.2021
by Slawek Kaplonski 25 Mar '21

25 Mar '21

Hi, Agenda for tomorrow's drivers meeting is at [1]. We have 2 new RFEs to discuss: * https://bugs.launchpad.net/neutron/+bug/1920065 - Automatic rescheduling of BGP speakers on DrAgents Patch for that is already proposed too https://review.opendev.org/c/openstack/neutron-dynamic-routing/+/780675 * https://bugs.launchpad.net/neutron/+bug/1921126 -[RFE] Allow explicit management of default routes Spec for that one is also proposed already https://review.opendev.org/c/openstack/neutron-specs/+/781475 Please read through them so we can discuss them tomorrow on the meeting. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[sdk][cli] OpenStackSDK/CLI Xena PTG
by Artem Goncharov 25 Mar '21

25 Mar '21

Hi, all The Xena PTG is scheduled for Apr 19-23 (15:00-17:00 UTC) We are organizing 2 hours in Wed 12 Apr for SDK/CLI project talks. Please add your PTG topics in the etherpad: https://etherpad.opendev.org/p/xena-ptg-sdk-cli <https://etherpad.opendev.org/p/xena-ptg-sdk-cli> P.S. And don’t forget to register for PTG participation Regards, Artem (gtema)

1 0

[horizon] [ptg] Xena PTG
by vishal manchanda 25 Mar '21

25 Mar '21

Hi everyone, As discussed in yesterday horizon weekly meeting and based on the doodle poll I have booked available slots for Horizon Xena PTG. Please find below the schedule for the same: Monday April 19, 14:00 - 17:00 UTC Tuesday April 20, 05:00 - 07:00 UTC and 16:00 - 17:00 UTC Wednesday April 21, 16:00 - 17:00 UTC I have also created Etherpad to collect topics for ptg discussion [1]. Feel free to add your topics. Please Let me know and Ivan in case you have any topics to discuss and above schedule doesn't work for you and We will see how to manage that. Thank you, Vishal Manchanda(irc: vishalmanchanda) [1] https://etherpad.opendev.org/p/xena-ptg-horizon-planning

1 0

[tripleo][ansible] Openstack Ansible collections (modules) Xena PTG
by Sagi Shnaidman 24 Mar '21

24 Mar '21

Hi, all The Xena PTG is scheduled for Apr 19-23 We are organizing 2 hours in Wed 12 Apr for Openstack Ansible collections project talks.[1] Please add your PTG topics in the etherpad: https://etherpad.opendev.org/p/xena-ptg-os-ansible-collections [1] https://opendev.org/openstack/ansible-collections-openstack Thanks -- Best regards Sagi Shnaidman

1 0

[kolla] Plan to deprecate and finally drop the chrony container
by Radosław Piliszek 24 Mar '21

24 Mar '21

Hello Fellow OpenStackers interested in Kolla's way. The topic should be pretty self-explanatory but let me provide the details and background. We are currently working on the Wallaby release and, since we are cycle-trailing, it is the right moment for us to reflect upon the various cleanup plans we had (and have). One of those is the drop of the chrony container. Chrony serves the time synchronization purpose which is required for healthy operations (I hope I don't need to explain this point further, time sync is essential for most services nowadays). It is still deployed by default by Kolla Ansible. The point is, most default (even minimal ones) host OS installations come with some form of NTP client, be it chrony, ntpd, timesyncd or whatever else there is. :-) This applies also to provisioning solutions like MAAS or Foreman. And Kolla Ansible actually stands on its head to detect and disable those to avoid multiple time sync daemons hell as it may disrupt normal operations (mostly due to having different time sources but also due to different compensation algorithms [or rather just their actual configuration] and the fact that each tool assumes to be the only source of time changes). We seem to have placed a relatively sane precheck to detect this but effective disabling may prove tricky, especially since it is very easy to get it installed at any point. The NTP client is considered a very basic service these days. It also seems that several users known to us do not use the chrony container and instead orchestrate timesync elsewhere. Hence, our current plan is to: 1) Deprecate the chrony container in the Wallaby cycle. - in Kolla Ansible this would mean also warning users that they use it and defaulting to not using it (with docs and a reno); docs would suggest an Ansible way to coordinate timesync - in Kolla docs and a reno 2) Drop the chrony container from both the places I know Kayobe (the "highest-level" member of the Kolla project) is planning to incorporate timesync handling by external Ansible role means so their users should be covered. Please let us know if you have any comments on the plan above. Or generally any suggestions you would like us to hear. Cheers! -yoctozepto

1 1