September 2021 - openstack-discuss

Fwd: DPDK+OVS with OpenStack
by Mark Wittling 23 Apr '25

23 Apr '25

Looking for someone who knows OpenStack with OpenVSwitch, and in addition to that, DPDK with OpenStack and OVS.I am using OpenStack Queens, with OpenVSwitch. The architecture I am using is documented here: https://docs.openstack.org/neutron/queens/admin/deploy-ovs-provider.htmlThe OVS I am using on the Compute Node, is compiled with DPDK, and I have enabled the datapath to netdev (DPDK) on br-prv (provider network bridge), and br-tun (tunneling bridge). But these two bridges, br-tun and br-prv, are patched into another OpenStack bridge, called br-int. I wasn’t actually sure about whether to tinker with this bridge, and wondered what datapath it was using.Then, I realized there is a parameter in the openvswitch_agent.ini file, which I will list here: # OVS datapath to use. 'system' is the default value and corresponds to the # kernel datapath. To enable the userspace datapath set this value to 'netdev'. # (string value) # Possible values: # system - <No description provided> # netdev - <No description provided> #datapath_type = system datapath_type = netdev So in tinkering with this, what I realized, is that when you set this datapath_type to system or netdev, it will adjust the br-int bridge to that datapath type.So here is my question. How can I launch a non-DPDK VM, if all of the bridges are using the netdev datapath type?Here is another question. What if one of the flavors don’t have the largepages property set on them? I assumed OpenStack would revert to a system datapath and not use DPDK for those VM interfaces. Well, I found out in testing, that is not the case. If you set all your bridges up for netdev, and you don’t set the property on the Flavor of the VM (largepages), the VM will launch, but it simply won’t work.Is there no way, on a specific Compute Host, to support both DPDK (netdev datapaths) and non-DPDK (system datapaths)?Either on a VM interface level (VM has one interface that is netdev DPDK and another that is system datapath non-DPDK)?Or on a VM by VM basis (VM 1 has 1 or more netdev datapath interfaces and VM 2 has 1 or more system datapath interfaces)?Am I right here? Once you set up a Compute Host for DPDK, it’s DPDK or nothing on that Compute Host? (edited)

3 2

How to config DPDK compute in OVN
by dangerzone ar 22 Apr '25

22 Apr '25

Hi....as far I understand ovs does support dpdk compute to be deployed. I'm referring to the guideline ' *https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html <https://docs.openstack.org/networking-ovn/queens/admin/dpdk.html>* ' ... It's too brief.. May I know how to config dpdk compute in OVN networking openstack? Hope someone could help what is the config to be done.... Currently compute is run without dpdk over openstack train ovn based. Please need some info on how I can proceed. Thank you

2 1

Live migration fails
by Szabo, Istvan (Agoda) 07 Aug '24

07 Aug '24

Hi, I have couple of compute nodes where the live migration fails with existing vms. When I quickly spawn a vm and try live migration it works so I assume shouldn't be a big problem with the compute node. However I have many existing vms where it fails with a servername not found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 ERROR nova.conductor.tasks.migrate [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Unable to find record for source node servername on servername: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Failed to compute_task_migrate_server: Compute host servername could not be found.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.605 227612 WARNING nova.scheduler.utils [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] [instance: 1517a2ac-3b51-4d8d-80b3-89a5614d1ae0] Setting instance to ACTIVE state.: ComputeHostNotFound: Compute host servername could not be found. /var/log/nova/nova-conductor.log:2021-04-21 14:47:12.672 227612 ERROR oslo_messaging.rpc.server [req-f4067a26-a233-4673-8c07-9a8a290980b0 dce35e6eceea4312bb0baa0510cef363 ca7e35079f4440c78bd9870724b9638b - default default] Exception during message handling: ComputeHostNotFound: Compute host am-osfecn-4025 Tried with this command: nova live-migration --block-migrate id. Any idea? Thank you. ________________________________ This message is confidential and is for the sole use of the intended recipient(s). It may also be privileged or otherwise protected by copyright or other legal rules. If you have received it by mistake please let us know by reply email and delete it from your system. It is prohibited to copy this message or disclose its content to anyone. Any confidentiality or privilege is not waived or lost by any mistaken delivery or unauthorized disclosure of the message. All messages sent to and from Agoda may be monitored to ensure compliance with company policies, to protect the company's interests and to remove potential malware. Electronic messages may be intercepted, amended, lost or deleted, or contain viruses.

5 16

[all] Dynamic Zuul results table in Gerrit 3
by Radosław Piliszek 04 Aug '22

04 Aug '22

Hello Fellow OpenStack and OpenDev Folks! TL;DR click on [3] and enjoy. I am starting this thread to not hijack the discussion happening on [1]. First of all, I would like to thank gibi (Balazs Gibizer) for hacking a way to get the place to render the table in the first place (pun intended). I have been a long-time-now user of [2]. I have improved and customised it for myself but never really got to share back the changes I made. The new Gerrit obviously broke the whole script so it was of no use to share at that particular state. However, inspired by gibi's work, I decided to finally sit down and fix it to work with Gerrit 3 and here it comes: [3]. Works well on Chrome with Tampermonkey. Not tested others. I hope you will enjoy this little helper (I do). I know the script looks super fugly but it generally boils down to a mix of styles of 3 people and Gerrit having funky UI rendering. Finally, I'd also like to thank hrw (Marcin Juszkiewicz) for linking me to the original Michel's script in 2019. [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/019051… [2] https://opendev.org/x/coats/src/commit/444c95738677593dcfed0cfd9667d4c4f0d5… [3] https://gist.github.com/yoctozepto/7ea1271c299d143388b7c1b1802ee75e Kind regards, -yoctozepto

5 7

[all] PyCharm Licenses Renewed till July 2021
by Swapnil Kulkarni 13 Jul '22

13 Jul '22

Hello, Happy to inform you the open source developer license for Pycharm has been renewed for 1 additional year till July 2021. Best Regards, Swapnil Kulkarni coolsvap at gmail dot com

8 13

Nova not updating to new size of an extended in-use / attached cinder volume (Ceph RBD) to guest
by Christian Rohmann 28 Jun '22

28 Jun '22

Hello Openstack-Discuss, after some digging I found out that cinder learned to resize attached (in-use) volumes quite a while ago with the introduction of the initial "extend 'in-use' volume" feature: https://review.opendev.org/c/openstack/cinder/+/454287/ The support was then extended to also cover Ceph RBD backed volumes with: https://review.opendev.org/c/openstack/nova/+/613039/ Since this is only about the cinder part, I was wondering if nova would ever find out and would actively rescan the device / volume and quintessentially present an increased new size to the guest. Apparently this is where a certain volume-extended event comes into play: https://review.opendev.org/c/openstack/nova/+/454322/ that is to be emitted by cinder when a volume had been extened. I then went head and tried this with the openstack cli (as Horizon does not seem to support / offer to resize in-use volumes. I run Openstack Train with Ceph RBD as storage. First I ran into an issue with the openstack cli (https://bugs.launchpad.net/cinder/+bug/1871759/comments/2 <https://bugs.launchpad.net/cinder/+bug/1871759/comments/2>) but using cinder extend $volumeid $newsize I was able to resize in-use volumes just fine. The only thing missing was the propagation to the guest. I played around with SCSI rescans, but that did not work, reboots also failed to do anyting in this regards. Stopping and starting the VM did work, but why having the capability to online-resize an attached volume when not being able to resize the guest volume as well? Apparently there seems to be an old and somewhat similar observation / bug with https://bugs.launchpad.net/nova/+bug/1369465 So I was simply wondering if this is expected to be working? Are there any special settings / options I need to set to enable this feature? Thanks and with kind regards Christian

4 9

[ops][nova][victoria] Power State = Suspended?
by DHilsbos＠performair.com 24 Jan '22

24 Jan '22

All; I had something unusual happen this morning; one of my VMs was showing "Suspended" under the Power State in the Horizon dashboard. I've never seen that. What does it mean? Any search that I do points me to a bunch of resources for Status Suspended. Thank you, Dominic L. Hilsbos, MBA Vice President - Information Technology Perform Air International Inc. DHilsbos(a)PerformAir.com www.PerformAir.com

4 8

[nova][dev] Revisiting qemu emulation where guest arch != host arch
by Apsey, Christopher 05 Jan '22

05 Jan '22

All, A few years ago I asked a question[1] about why nova, when given a hw_architecture property from glance for an image, would not end up using the correct qemu-system-xx binary when starting the guest process on a compute node if that compute nodes architecture did not match the proposed guest architecture. As an example, if we had all x86 hosts, but wanted to run an emulated ppc guest, we should be able to do that given that at least one compute node had qemu-system-ppc already installed and libvirt was successfully reporting that as a supported architecture to nova. It seemed like a heavy lift at the time, so it was put on the back burner. I am now in a position to fund a contract developer to make this happen, so the question is: would this be a useful blueprint that would potentially be accepted? Most of the time when people want to run an emulated guest they would just nest it inside of an already running guest of the native architecture, but that severely limits observability and the task of managing any more than a handful of instances in this manner quickly becomes a tangled nightmare of networking, etc. I see real benefit in allowing this scenario to run natively so all of the tooling that exists for fleet management 'just works'. This would also be a significant differentiator for OpenStack as a whole. Thoughts? [1] http://lists.openstack.org/pipermail/openstack-operators/2018-August/015653… Chris Apsey Director | Georgia Cyber Range GEORGIA CYBER CENTER 100 Grace Hopper Lane | Augusta, Georgia | 30901 https://www.gacybercenter.org

2 2

[stable][requirements][zuul] unpinned setuptools dependency on stable
by Balazs Gibizer 15 Dec '21

15 Dec '21

Hi, The latest setuptools (58.0) removed support for "use_2to3" [1] (deprecated since 46.2). Many OpenStack modules defines decorator==3.4.0 as lower constraints[2]. However, decorator 3.4.0 cannot be installed anymore with the latest setuptool as it depends on "use_2to3". On master, this can be solved easily by bumping the dependency to decorator 4.0.0. On stable/xena we can still solve it the same way with a new RC. But on older stable branches such solution might be against the stable policy as it would require a major bump on our dependencies. This issue is not limited to lower-constraints testing it just hit us there first. A similar break could happen in our upper constraints as well on old stable branches. The root of the problem is that we always use the latest setuptools in our CI testing even on old stable branches. Zuul's ensure-tox task[4] installs tox which installs the virtualenv package which bundles the latest setuptools package. This happens without applying any constraints. Then tox is used to install the project's dependencies under lower or upper constraints with the unconstrained setuptools available. During and after yesterday's nova meeting [3] we discussed possible solutions. Option 1: Bump the major version of the decorator dependency on stable. Pros: * easy to implement Cons: * might against the policy / breaks downstream packagers * needs to be done in each affected project[3] separately * not future proof. After a future setuptools release we can see a similar break with another of our dependencies. @Stable Cores: how do you feel about such bump? Option 2: Pin the setuptools version during tox installation Pros: * single solution for all affected projects * future proof against breaks introduced by future setuptools releases Cons: * complex change as it probably requires to extend the base task in zuul itself Option 3: turn off lower-constraints testing Pros: * easy to implement * some of us want to get rid of lower constraints anyhow Cons: * needs per project changes * not future proof against similar break affecting our upper constraints on old stable branches. Option 4: utilize pyproject.toml[6] to specify build-time requirements Unfortunately, I'm not sure if it is possible to restrict the maximum setuptools version with it As a side note I tried applying [tox]requires in tox.ini to restrict setuptools version[7] but It does not seem to take effect in our case[8]. @Stable Cores: what do you think what could be the way forward? Cheers, gibi [1] https://setuptools.pypa.io/en/latest/history.html#v58-0-2 [2] https://codesearch.openstack.org/?q=decorator%3D%3D3.4.0&i=nope&literal=nop… [3] https://meetings.opendev.org/irclogs/%23openstack-nova/%23openstack-nova.20… [4] https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-tox/tasks… [5] https://tox.readthedocs.io/en/latest/config.html#conf-requires [6] https://www.python.org/dev/peps/pep-0518/ [7] https://review.opendev.org/c/openstack/placement/+/810293/1/tox.ini [8] https://zuul.opendev.org/t/openstack/build/f78a0300c8734e7c8475309af1d2e1a4…

11 28

[all][sphinx] "rST localisation for language "foo" not found" error
by Stephen Finucane 24 Nov '21

24 Nov '21

We've seen issues in some projects, whereby attempting to build translations as part of the 'openstack-tox-docs' zuul job [1] can result in the following warning: Warning, treated as error: ../doc/source/contributor/contributing.rst::rST localisation for language "id" not found. Based on [2], this issue appears to have been introduced by a recent version bump of docutils, which was in turn introduced by the introduction of Sphinx 4.x. We don't have a resolution for this bug yet but we'll update when we do. Hopefully this email will improve the "Google'ability" of this bug. Stephen [1] https://opendev.org/openstack/openstackdocstheme/src/commit/08461c5311aa692… [2] https://lists.freebsd.org/pipermail/freebsd-ports/2021-April/120848.html

2 2

[docs] Double headings on every page
by Peter Matulis 04 Nov '21

04 Nov '21

Hi, I'm hitting an oddity in one of my projects where the titles of all pages show up twice. Example: https://docs.openstack.org/project-deploy-guide/charm-deployment-guide/wall… Source file is here: https://opendev.org/openstack/charm-deployment-guide/src/branch/master/depl… Does anyone see what can be causing this? It appears to happen only for the current stable release ('wallaby') and 'latest'. Thanks, Peter

5 6

[Nova][interop]
by Arkady Kanevsky 20 Oct '21

20 Oct '21

Sean and Nova team, Interop WG would like 20-30 minutes on Yaga PTG agenda to discuss latest interop guidelines for Nova. If you can point us to etherpad then we can add it to agenda. Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456

3 6

[all][tc] Yoga TC-PTG Planning
by Ghanshyam Mann 20 Oct '21

20 Oct '21

Hello Everyone, As you already know that the Yoga cycle virtual PTG will be held between 18th - 22nd October[1]. To plan the Technical Committee PTG sessions, please do the following: 1. Fill the below doodle poll as per your availability. Please fill it soon as deadline to book the slot is 21th July. - https://doodle.com/poll/6dfdmdfi4s8wc7cd 2. Add the topics you would like to discuss to the below etherpad. - https://etherpad.opendev.org/p/tc-yoga-ptg NOTE: this is not limited to TC members only; I would like all community members to fill the doodle poll and, add the topics you would like or want TC members to discuss in PTG. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023540.html -gmann

1 2

[tacker] Yoga vPTG planning
by yasufum 15 Oct '21

15 Oct '21

Hi everyone, The next vPTG will be held on 18-22 October as I shared at the previous IRC meeting [1]. Registration has already opened [2]. We've decided to have the next vPTG session on the same timeslots, 6-8 UTC, as previous for most of us join from India and APAC regions. I've prepared etherpad for the next vPTG [3]. If you have any suggestion, please add your topic on it. [1] http://lists.openstack.org/pipermail/openstack-discuss/2021-July/023540.html [2] https://openinfra-ptg.eventbrite.com/ [3] https://etherpad.opendev.org/p/tacker-yoga-ptg Thanks, Yasufumi

2 1

[dev][cinder] Consultation about new cinder-backup features
by Daniel de Oliveira Pereira 14 Oct '21

14 Oct '21

Hello everyone, We have prototyped some new features on Cinder for our clients, and we think that they are nice features and good candidates to be part of upstream Cinder, so we would like to get feedback from OpenStack community about these features and if you would be willing to accept them in upstream OpenStack. Our team implemented the following features for cinder-backup service: 1. A multi-backend backup driver, that allow OpenStack users to choose, via API/CLI/Horizon, which backup driver (Ceph or NFS, in our prototype) will be used during a backup operation to create a new volume backup. 2. An improved NFS backup driver, that allow OpenStack users to back up their volumes to private NFS servers, providing the NFS hostpath at runtime via API/CLI/Horizon, while creating the volume backup. Considering that cinder was configured to use the multi-backend backup driver, this is how it works: During a volume backup operation, the user provides a "location" parameter to indicate which backend will be used, and the backup hostpath, if applicable (for NFS driver), to create the volume backup. For instance: - Creating a backup using Ceph backend: $ openstack volume backup create --name <backup_name> --location ceph <volume_id> - Creating a backup using the improved NFS backend: $ openstack volume backup create --name <backup_name> --location nfs://my.nfs.server:/backups <volume_id> If the user chooses Ceph backend, the Ceph driver will be used to create the backup. If the user chooses the NFS backend, the improved NFS driver, previously mentioned, will be used to create the backup. The backup location, if provided, is stored on Cinder database, and can be seen fetching the backup details: $ openstack volume backup show <backup_name> Briefly, this is how the features were implemented: - Cinder API was updated to add an optional location parameter to "create backup" method. Horizon, and OpenStack and Cinder CLIs were updated accordingly, to handle the new parameter. - Cinder backup controller was updated to handle the backup location parameter, and a validator for the parameter was implemented using the oslo config library. - Cinder backup object model was updated to add a nullable location property, so that the backup location could be stored on cinder database. - a new backup driver base class, that extends BackupDriver and accepts a backup context object, was implemented to handle the backup configuration provided at runtime by the user. This new backup base class requires that the concrete drivers implement a method to validate the backup context (similar to BackupDriver.check_for_setup_error) - the 2 new backup drivers, previously mentioned, were implemented using these new backup base class. - in BackupManager class, the "service" attribute, that on upstream OpenStack holds the backup driver class name, was re-implemented as a factory function that accepts a backup context object and return an instance of a backup driver, according to the backup driver configured on cinder.conf file and the backup context provided at runtime by the user. - All the backup operations continue working as usual. Could you please let us know your thoughts about these features and if you would be open to adding them to upstream Cinder? If yes, we would be willing to submit the specs and work on the upstream implementation, if they are approved. Regards, Daniel Pereira

5 6

[TripleO] Issue in running Pre-Introspection
by Anirudh Gupta 13 Oct '21

13 Oct '21

Hi Team, I tried installing Undercloud using the below link: https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/deploym… I am getting the following error: (undercloud) [stack@undercloud ~]$ openstack tripleo validator run --group pre-introspection Selected log directory '/home/stack/validations' does not exist. Attempting to create it. +--------------------------------------+---------------------------------+--------+------------+-----------------+-------------------+-------------+ | UUID | Validations | Status | Host_Group | Status_by_Host | Unreachable_Hosts | Duration | +--------------------------------------+---------------------------------+--------+------------+-----------------+-------------------+-------------+ | 7029c1f6-5ab4-465d-82d7-3f29058012ce | check-cpu | PASSED | localhost | localhost | | 0:00:02.531 | | db059017-30f1-4b97-925e-3f55b586d492 | check-disk-space | PASSED | localhost | localhost | | 0:00:04.432 | | e23dd9a1-90d3-4797-ae0a-b43e55ab6179 | check-ram | PASSED | localhost | localhost | | 0:00:01.324 | | 598ca02d-258a-44ad-b78d-3877321cdfe6 | check-selinux-mode | PASSED | localhost | localhost | | 0:00:01.591 | | c4435b4c-b432-4a1e-8a99-00638034a884 | *check-network-gateway | FAILED* | undercloud | *No host matched* | | | | cb1eed23-ef2f-4acd-a43a-86fb09bf0372 | *undercloud-disk-space | FAILED* | undercloud | *No host matched* | | | | abde5329-9289-4b24-bf16-c4d82b03e67a | *undercloud-neutron-sanity-check | FAILED* | undercloud | *No host matched* | | | | d0e5fdca-ece6-4a37-b759-ed1fac31a10f | *ctlplane-ip-range | FAILED* | undercloud | No host matched | | | | 91511807-225c-4852-bb52-6d0003c51d49 | *dhcp-introspection | FAILED* | undercloud | No host matched | | | | e96f7704-d2fb-465d-972b-47e2f057449c |* undercloud-tokenflush | FAILED *| undercloud | No host matched | | | As per the validation link, https://docs.openstack.org/tripleo-validations/wallaby/validations-pre-intr… check-network-gateway If gateway in undercloud.conf is different from local_ip, verify that the gateway exists and is reachable Observation - In my case IP specified in local_ip and gateway, both are pingable, but still this error is being observed ctlplane-ip-range¶ <https://docs.openstack.org/tripleo-validations/wallaby/validations-pre-intr…> Check the number of IP addresses available for the overcloud nodes. Verify that the number of IP addresses defined in dhcp_start and dhcp_end fields in undercloud.conf is not too low. - ctlplane_iprange_min_size: 20 Observation - In my case I have defined more than 20 IPs Similarly for disk related issue, I have dedicated 100 GB space in /var and / Filesystem Size Used Avail Use% Mounted on devtmpfs 12G 0 12G 0% /dev tmpfs 12G 84K 12G 1% /dev/shm tmpfs 12G 8.7M 12G 1% /run tmpfs 12G 0 12G 0% /sys/fs/cgroup /dev/mapper/cl-root 100G 2.5G 98G 3% / /dev/mapper/cl-home 47G 365M 47G 1% /home /dev/mapper/cl-var 103G 1.1G 102G 2% /var /dev/vda1 947M 200M 747M 22% /boot tmpfs 2.4G 0 2.4G 0% /run/user/0 tmpfs 2.4G 0 2.4G 0% /run/user/1000 Despite setting al the parameters, still I am not able to pass pre-introspection checks. *"NO Host Matched" *is found in the table. Regards Anirudh Gupta

2 3

[kuryr] Virtual PTG October 2021
by Maysa De Macedo Souza 12 Oct '21

12 Oct '21

Hello, I booked the following slots for Kuryr during the Yoga PTG: Oct 19 7-8 UTC and Oct 22 13-14 UTC. If you have any topic ideas you would like to discuss, please include them in the etherpad[1], also it would be interesting to include your name there if you plan to attend any Kuryr session. See you on the next PTG. Cheers, Maysa. [1] https://etherpad.opendev.org/p/kuryr-yoga-ptg

1 2

Edge sessions at the upcoming PTG
by Ildiko Vancsa 11 Oct '21

11 Oct '21

Hi, I’m reaching out to you to share the agenda of the OpenInfra Edge Computing Group that we put together for the upcoming PTG. I would like to invite everyone who is interested in discussing edge challenges and finding solutions! We summarized our plans for the event in a short blog post to give some context to each of the topic that we picked to discuss in details. We picked key areas like security, networking, automation and tools, containers and more: https://superuser.openstack.org/articles/the-project-teams-gathering-is-com… Our etherpad for the sessions: https://etherpad.opendev.org/p/ecg-ptg-october-2021 Please let me know if you have any questions about the agenda or topics. Thanks and Best Regards, Ildikó

1 2

[Swift][Interop] PTG time
by Arkady Kanevsky 06 Oct '21

06 Oct '21

Swift team, Interop team would like time on Yoga PTG Monday or Tu between 21-22 UTC to discuss Interop guideline coverage for Swift. Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456

3 5

[masakari] Proposal to cancel weekly meetings
by Radosław Piliszek 06 Oct '21

06 Oct '21

Dears, Due to low attendance and the current schedule being uncomfortable for me, I propose to cancel the weekly meetings and suggest we coordinate via this mailing list and do ad-hoc chats on IRC as I'm simply lurking there most of the time and answering the messages. Kind regards, -yoctozepto

1 1

[ironic][molteniron][qa] Anyone still using MoltenIron?
by Julia Kreger 05 Oct '21

05 Oct '21

Out of curiosity, is anyone still using MotltenIron? A little historical context: It was originally tooling that came out of IBM to reserve physical nodes in a CI cluster in order to perform testing. It was never intended to be released as it was tooling *purely* for CI job usage. The reason I'm asking is that the ironic team is considering going ahead and retiring the repository. Thanks! -Julia

3 3

Help with eventlet 0.26.1 and dnspython >= 2
by Thomas Goirand 04 Oct '21

04 Oct '21

Hi, As you may know, eventlet is incompatible with dnspython >= 2.0.0.0rc1. See [1] for the details. However, Debian unstable has 2.0.0. Would there be some good soul willing to help me fix this situation? I would need a patch to fix this, but I'm really not sure how to start. Cheers, Thomas Goirand (zigo)

8 15

[TripleO] Gate blocker - please hold rechecks
by Ronelle Landy 01 Oct '21

01 Oct '21

Hello All, We have a gate blocker for tripleo at: https://bugs.launchpad.net/tripleo/+bug/1945682 This tox error is impacting tox jobs on multiple tripleo-related repos. A resolution is being worked on by infra. Please hold rechecks if you are rechecking for this failure. Please note that third-party OVB jobs are also down - following an update to DNS on a network (related bug: https://bugs.launchpad.net/tripleo/+bug/1945678) We will update this list when the error is cleared. Thanks for your patience!

2 3

[oslo] oslo.metrics package for Fedora
by Hirotaka Wakabayashi 01 Oct '21

01 Oct '21

Hello Oslo Team, I am a Fedora packager. I want to package oslo.metrics for Fedora because my package uses oslo.messaging that requires oslo.metrics as you know. oslo.messaging package repository already exists in Fedora. I will take over it from the former package maintainer. the oslo.metrics repository doesn't existso I need to make it. If any concerns with it, please reply. I can update the version as soon as the new version releases by using Fedora's release monitoring system. Thanks in advance, Hirotaka Wakabayashi

4 4

[cinder][nova][requirements] RFE requested for os-brick
by Brian Rosmaita 30 Sep '21

30 Sep '21

Hello Requirements Team, The Cinder team recently became aware of a regression in os-brick that prevents Nova from attaching NVMe volumes [0]. A fix has been merged to master (yoga development) [1] and has been backported to stable/xena [2]. We've proposed a release of os-brick 5.0.1 from stable/xena [3], and are petitioning for an Requirements Freeze Exception to include os-brick 5.0.1 in the xena release. Thank you for your attention to this matter. [0] https://bugs.launchpad.net/os-brick/+bug/1945323 [1] https://review.opendev.org/c/openstack/os-brick/+/811944 [2] https://review.opendev.org/c/openstack/os-brick/+/812010 [3] https://review.opendev.org/c/openstack/releases/+/812048

2 1

[neutron] Drivers meeting agenda - 01.10.2021
by Lajos Katona 30 Sep '21

30 Sep '21

Hi, The agenda for tomorrow's drivers meeting is at [1]. We have 1 RFE to discuss: * https://bugs.launchpad.net/neutron/+bug/1885261 Add stateless firewall support to OVS firewall ** It is Liu who would like to implement this, see logs of last team meeting at [2] ** Liu can't participate as there's some celebration in China, and he is on holiday, but still we can discuss it to have better understanding. [1] https://wiki.openstack.org/wiki/Meetings/NeutronDrivers#Agenda [2] https://meetings.opendev.org/meetings/networking/2021/networking.2021-09-28… See you at the meeting tomorrow. Lajos Katona (lajoskatona

1 0

Open stack Swift
by Midhunlal Nb 30 Sep '21

30 Sep '21

Hi team, I am using Openstack rocky setup with 1 controller node 2 compute node and 1 sStorage node,Now i am planning to configure a storage backup service Swift. Where do I need to configure swift,It needs an extra Node?Please explain the requirements. Thanks & Regards Midhunlal N B +918921245637

3 2

how dpdk improve guest performance
by Satish Patel 30 Sep '21

30 Sep '21

Folks, If I am running dpdk in my compute nodes then sure it will reduce kernel overhead but how does it help to improve guest network performance? Inside guest vm it's going to use kernel to copy packet to userspace. in my example we run high performance haproxy which handles 300k to 500k connections but I have noticed performance is poor and lots of packet drops. If I run the same workload on SRIOV vms it works fine and I notice less kernel context switching inside the guest VM. I think when you run DPDK in compute then it only help compute but not your guest network performance. How do I improve guest performance also with dpdk based compute?

1 0

[nova] WSGI Apache vs Eventlet HTTP Servers
by Ammad Syed 30 Sep '21

30 Sep '21

Hi, I was reviewing below document. https://docs.openstack.org/nova/wallaby/user/wsgi.html Is apache2 wsgi provide better performance then eventlet http servers for nova api ? If yes, can you guys please share the sample or reference apache-nova-api file for wsgi ? - Ammad

1 0

[GPU] Request for suggestions for OpenStack deployment
by Jay See 30 Sep '21

30 Sep '21

Hi, We are planning to buy some servers with GPU cards from DELL(R740/R750)/HPE/Supermicro with NVIDIA A100 40GB (GPU). I have the following questions: 1. What is the best ratio for Memory to GPU? or It doesn´t matter. 2. Anyone using A100? A100 has any known issues with OpenStack deployment? 3. If you have set up with similar server with GPU, could you share the server configuration? It will help us in deciding / identifying the components we have missed. Thanks in advance. Regards, ~ Jayachander.

1 0

DC DR high availability - Snapshot taking feature and user experience
by KK CHN 30 Sep '21

30 Sep '21

List, I am in the process of setting up a DC and DR sites with openstack(ussuri, qemu kvm with debian as base OS). I would like to know what all are the options in DC and DR sites for High availability . What the architecture methods to follow at DC and DR so that, If VM crashes at one physical machine at DC ( for example 3 controllers in DC), or Host machine crashes( RAM, Hardware failure etc. ) / machine power cable detached accidentally so that no services down / VMs and applications not down. How can achieve this ? share the best practices here. Also If I take snapshots of the running VMs ( what will the user experience ? will they in freeze / logged out from applications right now they are logged in ? ) . Can we avoid the service unavailable for users while taking snapshots ? To DR we are rsyncing these snapshots after converting each VM image to qcow2 then rsyncing . ( All these we are performing on Controller node . Generally 100 GB VM will output 16 GB qcow2 image for example. ) its taking 2 minutes for snapshot creation and 20 to 30 Minutes for qcow2 conversion . ) How many snapshots and conversion to qcow2( qemu-img convert) can be performed on this controller machine where we performing this operation . ( Can we apply parallel processing for this and how ? ) . Seet he controller specs where we perform this. The controller spec ( 20 core CPUs with 1.5 TB RAM total 160 processors ) Then copying to DR ( which is 200 KM away from our DC takes 4 minutes with rsync). Then we populate each VM with these copied qcow2 images and attaching the Network ips and nat there. Is this the practice or any other good way to perform this for a robust DC and DR setup . Kindly share your thoughts. Regards, Kris

1 0

[all][tc] Technical Committee next weekly meeting on Sept 30th at 1500 UTC
by Ghanshyam Mann 29 Sep '21

29 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 30th at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 29th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

DPDK Poor performance
by Satish Patel 29 Sep '21

29 Sep '21

Folk, I have deployed DPDK on one of my compute nodes to reply to my sriov deployment which doesn't support bonding. This is what I am doing to do a performance benchmark. Spun up VM with 8 vCPU and 8GB memory with single virtual nic. Then install nuttcp tool and start nuttcp -S (server mode) and then use Physical server outside my lab and use following command to blast some traffic nuttcp -T 30 -i -u -w4m -R 2G 10.69.7.130 I found 120kpps rate working fine but after that I noticed a packet drop on nuttcp. Here is the full details of my deployment and load-test result - https://paste.opendev.org/show/809675/ Small confusion related hugepage - I am seeing following in /dev/hugepages/ I believe rtemap_0 created by DPDK root@ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/rtemap_0 -rw------- 1 root root 1073741824 Sep 29 02:28 /dev/hugepages/rtemap_0 Why is the qemu instance file empty? Does that mean it's not using a huge page for dpdk performance? root@ovn-lab-comp-dpdk-1:~# ls -l /dev/hugepages/libvirt/qemu/1-instance-00000045/ total 0 I have hw:mem_page_size='large' setting in flavor so it should use hugepage. am i missing something?

1 0

[largescale-sig] Next meeting: Sept 29th, 15utc
by Thierry Carrez 29 Sep '21

29 Sep '21

Hi everyone, The Large Scale SIG meeting is back this Wednesday in #openstack-operators on OFTC IRC, at 15UTC. You can doublecheck how that time translates locally at: https://www.timeanddate.com/worldclock/fixedtime.html?iso=20210929T15 A number of topics have already been added to the agenda, including discussing the final details of our next OpenInfra.Live show! Feel free to add other topics to our agenda at: https://etherpad.openstack.org/p/large-scale-sig-meeting Regards, -- Thierry Carrez

1 1

[kolla] Yoga PTG
by Michał Nasiadka 29 Sep '21

29 Sep '21

Hola Koalas! We are approaching the time of the next PTG. Kolla Yoga etherpad is available and initially populated[1], feel free to add any topics that shall be discussed and register your planned attendance. Don't forget to register! [2]. [1] https://etherpad.opendev.org/p/kolla-yoga-ptg [3] https://openinfra-ptg.eventbrite.com Best regards, Michal Nasiadka

1 0

[cinder] Bug deputy report for week of 2021-29-09
by Sofia Enriquez 29 Sep '21

29 Sep '21

This is a bug report from 2021-15-09 to 2021-29-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/os-brick/+bug/1943977 "os_bricks fails to rerty "iscsiadm -m session" when iser_use_multipath and iscsi_use_multipath are set in nova.conf". Assigned to Sophie Huang. - https://bugs.launchpad.net/os-brick/+bug/1945323 "Can't attach NVMe volume" Low - https://bugs.launchpad.net/cinder/+bug/1945169 " [swift] volume backup's status keep in 'restoring' and Lost connection to MySQL server during query". Assigned to silphon zhang. Wishlist - https://bugs.launchpad.net/cinder/+bug/1945249 '[charm] no upgrade path to cinder-lvm subordinate'. Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[Cinder] [Nova] [Interop] "compute-volume-list" API guideline
by Vida Haririan 29 Sep '21

29 Sep '21

Hi Cinder and Nova teams, The Interop team is reviewing all related open stories to determine next steps. We researched the current status of 1638112:“os-volumes should be flagged for removal in the compute API guidelines“ [1] and noticed that "compute-volume-list" was removed in 2017.01 guidelines [2][3]. However, two related tests still remain in tempest [4][5] and we are reaching out to determine if the API is still in use or has been moved/renamed. Please let us know if you have any questions. [1] https://storyboard.openstack.org/#!/story/1638112 [2] https://opendev.org/osf/interop/src/commit/dd7ae2660de5daa82ab4fe080e875d63… [3] https://opendev.org/osf/interop/src/branch/master/guidelines/next.json [4] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f2… [5] https://opendev.org/openstack/tempest/src/commit/ae41052a51f5dbb748eb6bf4f2… Thank you, -- *Vida Haririan* Senior Software Quality Engineer Red Hat EMEA <https://www.redhat.com/>

2 1

[announce][debian][xena] general availability of OpenStack Xena in Debian
by Thomas Goirand 29 Sep '21

29 Sep '21

Hi! It's my pleasure to announce the general availability of OpenStack Xena in Debian. Indeed, I've just finished uploading everything to Debian Experimental at the beginning of this week, and was able yesterday, to pop my first VM over on top of Xena. This marks the first readiness of the release, from the Debian point of view (even though there probably will be things to fix, like after addressing https://bugs.launchpad.net/oslo.policy/+bug/1945336 for example, and uploading the final release to replace the RCs when that becomes available). The Bullseye backports are available the usual way, for example using extrepo: apt-get install extrepo extrepo enable openstack_xena apt-get update ... or directly setting-up the http://bullseye-xena.debian.net repository the usual way. I'm currently in the process of uploading all of Xena to Debian Unstable (but please bare with me: there's 225+ packages in this update). Note that Xena can already be installed by OCI [1]. Please report any issue you may find, on OCI or on the Debian packages, through the Debian BTS as usual. Cheers, Thomas Goirand (zigo) [1] https://salsa.debian.org/openstack-team/debian/openstack-cluster-installer

1 0

[kolla][neutron][victoria] metering agent on compute nodes
by Tomasz Rutkowski 29 Sep '21

29 Sep '21

Hi All, I have issue with neutron-metering-agent - on network node it works ok, however on compute nodes the list of routers is constantly empty (and l3-agent works normal)... how to debug why outside network node the metering-agent does not recieve list of routers? with best regards -- Tomasz Rutkowski

2 2

[all] Gate status: All devstack based jobs are failing for placement url issue in devstack
by Ghanshyam Mann 28 Sep '21

28 Sep '21

Hello Everyone, All the devstack based jobs are failing with placement URL issue in devstack. The root cause is not known yet. You will see the below error in devstack/summary log- + ./stack.sh:main:1455 : /usr/local/bin/nova-status --config-file /etc/nova/nova.conf upgrade check Placement logs: INFO placement.requestlog [None req-3ad7ab8b-7c85-477a-b156-f5ca652a38cc service placement] 10.208.226.48 "GET /placemen//" status: 404 len: 162 microversion: 1.0 Please do not recheck until the issue is fixed. -gmann

5 8

OpenInfra Live - September 30th at 9am CT
by Erin Disney 28 Sep '21

28 Sep '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenStack and Ceph communities. Ceph is a highly scalable distributed-storage open source solution offering object, block, and file storage. Join us as various Community members discuss the basics, ongoing development and integration of OpenStack with Ceph. Date and time: September 30th at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=zJVoleSpSOk LinkedIn: https://www.linkedin.com/video/event/urn:li:ugcPost:6846862917225320448/ Facebook: https://www.facebook.com/104139126308032/posts/4364096940312208/ WeChat: Recording will be posted on OpenStack WeChat after the live stream Speakers: Kendall Nelson (OpenInfra Foundation) John Fulton (Red Hat) Francesco Pantano (Red Hat) Tom Barron (Red Hat) Loan Harrouin (Societe Generale) Pawel Sadowski (OVH) Mohammed Naser (VEXXHOST) Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ Thanks, Erin Erin Disney Event Marketing Open Infrastructure Foundation

1 0

Issue on accessing object storage containers from outside
by Michel Niyoyita 28 Sep '21

28 Sep '21

Dear all I have created containers and upload objects into containers . I can access these containers and objects trough horizon dashboard but once make them public I can not reach them through the provided link . If any one have an idea on how to access containers and their object using the link from outside he can help and guide. I deployed openstack wallaby using kolla-ansible and ceph pacific using ansible , both are using ubuntu 20.04 as OS Michel

3 4

[openstack-qa] opendev.org mirrors
by Eduardo Santos 28 Sep '21

28 Sep '21

Hi folks, On US-based servers, a DevStack deploy takes 15 minutes. When deploying it from Brazil (where I live) however, it takes more than an hour. Examining the stack.sh logs, git_timed appears to be the culprit. I'm using a 250 Mbps connection. Do opendev.org mirrors exist for other regions other than the US? If not, has this been discussed yet? I believe this is more of a financial discussion othet than technical, but I'm curious if other users have raised this question before. I'm not sure if openstack-qa is the most appropriate flag. If not, please feel free to correct. Thanks!

3 3

Creating multiple servers with a shared volume
by open infra 28 Sep '21

28 Sep '21

Hi, I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM. Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later? Regards, Danishka

3 5

[Tripleo] Support of Alma/Rocky Linux
by Anirudh Gupta 28 Sep '21

28 Sep '21

Hi Team As per the document, the supporting OS mentioned in the documentation is only Centos Stream and No Stream version (as of now) Since Centos has officially stopped releasing new versions and looking for EOL. I Just wanted to check if there is a possibility that we can install Tripleo on a specific version of Alma Linux or Rocky Linux Regards Anirudh Gupta

2 3

RE: Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 28 Sep '21

28 Sep '21

Hi, I’ve turned off „central_logging” and now I have only several errors like below on all nodes (3-7 per day) 0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” On compute node, which had errors like this : /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag=„openstack_python" I’ve restarted fluentd container on that node and now it’s not showing errors like this anymore. So what’s the problem? Performance of Monasca/Kibana/Elasticsearch? But the node (telemetry) on which this stack is running is not overloaded - 20-30% CPU (24 cores on that node) , 40% RAM (there is 64GB), a lot of space on SSD. Best regards Adam Tomas > Hi, > after some time of using newly deployed kolla-ansible I have fluentd errors: > > on all controllers, storage and network nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” > > > on compute nodes: > > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" > /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” > > and on monitoring node: > > 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> > 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" > 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace > 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" > 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" > 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> > >

2 1

key_repository does not exist or Keystone does not have sufficient permission to access it [kolla-ansible][keystone]
by Adam Tomas 28 Sep '21

28 Sep '21

Hi, some next problem with my kolla-ansible deployment. On controller nodes I have (several times per hour) error messages like below: ERROR keystone.common.fernet_utils [req-386e223e-5f08-4959-bd0b-3dbbfd3b534f 73c7a9a7175e46e9971181963267225b efa5d1ce80324476b4ce189686dda2d1 - default default] Either [None] key_repository does not exist or Keystone does not have sufficient permission to access it: /etc/keystone/credential-keys/ I’ve checked - such path doesn’t exist inside keystone containers. I found bug #1863643 (https://bugs.launchpad.net/kolla/+bug/1863643 <https://bugs.launchpad.net/kolla/+bug/1863643>) but I don’t see any progress on it nor information that victoria is not affected (so I assume it’s the same bug and Victoria is affected?). What should be done to get rid of this errors? Best regards Adam Tomas

1 0

[cinder] reminder: this week's meeting in video+IRC
by Brian Rosmaita 27 Sep '21

27 Sep '21

Quick reminder that this week's Cinder team meeting on Wednesday 29 September, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-xena-meetings cheers, brian

1 0

[qa] Office Hour Sep 28th is cancelled
by Martin Kopec 27 Sep '21

27 Sep '21

Hi everyone, the next QA Office Hour on September 28th at 2pm UTC is cancelled due to a public holiday in my country and I won't be able to moderate it. Thank you for your understanding, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA

1 0

[OSSA-2021-002] Nova: Open Redirect in noVNC proxy (CVE-2021-3654)
by Jeremy Stanley 27 Sep '21

27 Sep '21

=========================================== OSSA-2021-002: Open Redirect in noVNC proxy =========================================== :Date: July 29, 2021 :CVE: CVE-2021-3654 Affects ~~~~~~~ - Nova: <21.2.3, >=22.0.0 <22.2.3, >=23.0.0 <23.0.3 Description ~~~~~~~~~~~ Swe Aung, Shahaan Ayyub, and Salman Khan with the Monash University Cyber Security team reported a vulnerability affecting Nova's noVNC proxying implementation which exposed access to a well-known redirect behavior in the Python standard library's http.server.SimpleHTTPRequestHandler and thus noVNC's WebSockifyRequestHandler which uses it. By convincing a user to follow a specially-crafted novncproxy URL, the user could be redirected to an unrelated site under control of the attacker in an attempt to convince them to divulge credentials or other sensitive data. All Nova deployments with novncproxy enabled are affected. Errata ~~~~~~ The initial fix did not take into account the possibility of bypass using exactly three slashes. This update provides a more thorough revised fix for the issue. The affected versions list has been updated to indicate versions expected to include the newer solution. Patches ~~~~~~~ - https://review.opendev.org/791807 (Train) - https://review.opendev.org/806629 (errata 1) (Train) - https://review.opendev.org/791806 (Ussuri) - https://review.opendev.org/806628 (errata 1) (Ussuri) - https://review.opendev.org/791805 (Victoria) - https://review.opendev.org/806626 (errata 1) (Victoria) - https://review.opendev.org/791577 (Wallaby) - https://review.opendev.org/805818 (errata 1) (Wallaby) - https://review.opendev.org/791297 (Xena) - https://review.opendev.org/805654 (errata 1) (Xena) Credits ~~~~~~~ - Swe Aung from Monash University Cyber Security team (CVE-2021-3654) - Shahaan Ayyub from Monash University Cyber Security team (CVE-2021-3654) - Salman Khan from Monash University Cyber Security team (CVE-2021-3654) References ~~~~~~~~~~ - https://launchpad.net/bugs/1927677 - https://bugs.python.org/issue32084 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-3654 Notes ~~~~~ - The stable/train branch is under extended maintenance and will receive no new point releases, but a patch for it is provided as a courtesy. OSSA History ~~~~~~~~~~~~ - 2021-09-27 - Errata 1 - 2021-07-29 - Original Version -- Jeremy Stanley

1 0