October 2019 - openstack-discuss - lists.openstack.org

[ceilometer] The correct way to change metric granularity time?
by Eddie Yen 25 Oct '19

25 Oct '19

Hi We're using Ceilometer with Gnocchi in Rocky release. And we're using cpu_util variable to doing auto scaling on Heat. We known that ceilometer will create its own archive policy in gnocchi, and default value of granularity is 5 minutes. We want change ceilometer's granularity to a minute. Here's our workaround. 1. Edit pipeline.yaml. Since gnocchi has default policy that granularity = 1 second, we change "gnocchi://" to "gnocchi://?archive_policy=high" 2. Edit ceilometer.conf & polling.yaml, change both evaluation_interval (ceilometer.conf) & interval (polling.yaml) to 60. 3. Restart all ceilometer services We tested this workaround but found that it would not work sometimes. And found that the first detection triggered every "o'clock" when created auto scaling at first time. If we want to shrink the granularity time, is our workaround correct? And how is the correct way to do this if not? Many thanks, Eddie.

1 0

[release] Release countdown for week R-28
by Sean McGinnis 25 Oct '19

25 Oct '19

Welcome back to the release countdown emails! These will be sent at major points in the Ussuri development cycle, which should conclude with a final release on May 13, 2020. Development Focus ----------------- At this stage in the release cycle, focus should be on planning the Ussuri development cycle, assessing Ussuri community goals and approving Ussuri specs. General Information ------------------- Please note that the Ussuri cycle will be slightly longer than past cycles (30 weeks). In case you haven't seen it yet, please take a look over the schedule for this release: https://releases.openstack.org/ussuri/schedule.html By default, the team PTL is responsible for handling the release cycle and approving release requests. This task can (and probably should) be delegated to release liaisons. Now is a good time to review release liaison information for your team and make sure it is up to date: https://opendev.org/openstack/releases/src/branch/master/data/release_liais… By default, all your team deliverables from the Train release are continued in Ussuri with a similar release model. If you intend to drop a deliverable, or modify its release model, please do so before the ussuri-1 milestone by proposing a change to the deliverable file at: https://opendev.org/openstack/releases/src/branch/master/deliverables/ussuri Upcoming Deadlines & Dates -------------------------- Forum+PTG at Shanghai summit: November 4 Ussuri-1 milestone: December 12 (R-22 week)

1 0

[cyborg] Nominating Chenke as core reviewer
by Nadathur, Sundar 25 Oct '19

25 Oct '19

Hello, Chenke has contributed substantially to Cyborg [1], esp. in the Train release, and has been an active participant in the community. I would like to acknowledge his significant past and ongoing contributions [2] and enthusiasm, and nominate him as a core reviewer for Cyborg. Please provide any feedback on his nomination by Oct 31, 2019. If there are no objections, his nomination will be made effective on Nov 1, 2019. [1] https://www.stackalytics.com/?module=cyborg-group&user_id=chenker [2] https://review.opendev.org/#/q/project:openstack/cyborg+owner:chenker Regards, Sundar

4 3

[charms] OpenStack Charms 19.10 release is now available
by Peter Matulis 25 Oct '19

25 Oct '19

The OpenStack Charms team is thrilled to announce the 19.10 charms release, introducing support for OpenStack Train and Ceph Nautilus on Ubuntu 18.04 LTS (via UCA) and Ubuntu 19.10. This release also brings several new and valuable features to the existing OpenStack Charms deployments for Queens, Rocky, Stein, and many other stable combinations of Ubuntu + OpenStack. Please see the Release Notes for full details: https://docs.openstack.org/charm-guide/latest/1910.html == Highlights == * OpenStack Train OpenStack Train is now supported on Ubuntu 18.04 LTS (via UCA) and Ubuntu 19.10. * Policy Overrides A new Policy Overrides feature provides operators with a mechanism to override policy defaults on a per-service basis. * Ceph Nautilus The Nautilus release of Ceph is now supported, in conjunction with OpenStack Train. * Ceph placement group autotuning In Ceph Nautilus, the autotuning of placement groups is now supported. * Neutron port forwarding The Neutron port forwarding extension can now be optionally enabled for OpenStack Rocky and later. * Migration to FQDN for agent registration When deploying OpenStack Stein or newer, the Nova Compute agent and Neutron agents will now use a fully qualified domain name (FQDN) when registering with the API services. * Ceph RADOS Gateway tenant namespacing The ceph-radosgw charm now supports deployment with tenant namespaces. * New charm: Placement There is a new charm for the placement API: the 'placement' charm. The new charm must be deployed and related to the nova-cloud-controller charm for OpenStack Train deployments. This will affect Stein to Train upgrades. * New charm: Cinder integration with Pure Storage array There is a new subordinate charm that can be used to integrate Cinder with a Pure Storage array: the 'cinder-purestorage' charm. * nova-cloud-controller: instance migration - DNS caching is now the default The caching of DNS lookups of the nova-compute units is now the default behaviour. == OpenStack Charms team == The OpenStack Charms team can be contacted on the #openstack-charms IRC channel on Freenode. The team will be at the Open Infrastructure Summit and PTG events in Shanghai (November 4-8, 2019). == Thank you == Massive appreciation of the below 59 charm contributors who squashed 41 bugs, enabled support for a new release of OpenStack, improved documentation, and added compelling new functionality! Frode Nordahl Chris MacNaughton Corey Bryant Ryan Beisner Camille Rodriguez David Ames Alex Kavanagh Liam Young James Page Dmitrii Shcherbakov Rodrigo Barbieri Peter Matulis Sahid Orentino Ferdjaoui Edward Hope-Morley Tytus Kurek Ghanshyam Mann Jorge Niedbalski Natalia Litvinova Tiago Pasqualini Nicolas Pochet Narinder Gupta exsdev Andrea Ieri Andreas Jaeger Eduardo Sousa Zachary Zehring Trent Lloyd Mike Wilson David Coronel Stamatis Katsaounis Dan Ackerson Hua Zhang Dongdong Tao Ian Wienand Felipe Reyes Ramon Grullon Joe Guo Erlon R. Cruz George Kraft Alvaro Uria Peter Sabaini Michael Skalka Nikolay Vinogradov Jose Delarosa melissaml Seyeong Kim Mark S. Maglana Nobuto Murata Andrew McLeod Frank Kloeker Tim Burke Cory Johns Marian Gasparovic sunnyve Pete Vander Giessen Ryan Farrell Levente Tamas Alexander Litvinov Marcelo Subtil Marcal -- OpenStack Charms Team

1 0

[api sig] office hour cancelled for today
by Michael McCune 24 Oct '19

24 Oct '19

due to schedule conflicts we are short on staff for the office hour today and it will be cancelled. peace o/

1 0

[nova] Is config drive data stored in ephemeral storage?
by Jean-Philippe Méthot 24 Oct '19

24 Oct '19

Hi, We currently use Ceph for storage on our Openstack cluster. We set up a pool to use for ephemeral nova storage, but we never actually use it, as we prefer to use cinder block devices. However, we notice that objects are being created inside that pool. Could it be that when you use config drives, the actual config drive is created inside nova’s ephemeral storage instead of on the compute node’s disk? Best regards, Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.

3 2

[qa] [all] Opinion on dropping the py2.7 support from Tempest & Tempest plugin
by Ghanshyam Mann 24 Oct '19

24 Oct '19

Hello Everyone, We are in Ussuri development cycle which is planned to drop the py2.7 support[1]. I was holding and planning to support the py2.7 in Tempest due to its branchless model[2]. My main point to keep py2 support is if any users running the Tempest on py27 env cloud and not in virtual env then they can keep running in the same way. But this might be just me overthinking on this usage. What happens if we drop py2.7 from Tempest: * Users with the above case have the way to install the latest Tempest on virtual env of py3. or use the Tempest tag if they do not need latest Tempest. * No change in users using Tempest on py3 or in py3 virtual env. * Upstream testing of master or stable branch is no issue as we install the Tempest in virtual env. Tempest in py3 venv can always test the py2.7 jobs. * other than that no change in Tempest usage or at least it would not break anything. Why we cannot keep the support: There is no big cost of supporting the py2.7 in Tempest itself but that require Temepst dependency (OpenStack lib like Oslo and non-openstack maintained lib [3]) to keep supporting the py2.7 which is not feasible. Other solution: One way is to cut the Tempest stable branch and keep the py2.7 support there with eligible backport from Tempest master which is py3 only. But I would say, QA team has no bandwidth to do so. if anyone wants to maintain that then we can discuss this option in more detail. I have given a second thought on this and now ok to drop the py2.7 from Tempest by considering all the above points. Please reply if any disagreement on this or add if anything I missed to consider. NOTE: Tempest includes its plugins also. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.… [2] https://review.opendev.org/#/c/681203/ [3] https://opendev.org/openstack/tempest/src/commit/7fdd39c6dbde37bccd419c4037… -gmann

7 12

[ironic][release][stable] Ironic Train release can be broken due to entry in driver-requirements.txt
by Richard.Pioso＠dell.com 24 Oct '19

24 Oct '19

Hi, The Ironic Train release can be broken due to an entry in its driver-requirements.txt. driver-requirements.txt defines a dependency on the sushy package [1] which can be satisfied by version 1.9.0. Unfortunately, that version contains a few bugs which prevent Ironic from being able to manage Dell EMC and perhaps other vendors' bare metal hardware with its Redfish hardware type (driver). The fixes to them [2][3][4] were merged into master before the creation of stable/train. Therefore, they are available on stable/train and in the last sushy release created during the Train cycle, 2.0.0, the only other version which can satisfy the dependency today. However, consumers -- packagers, operators, and users -- could, fighting time constraints or lacking solid visibility into Ironic, package or install Ironic with sushy 1.9.0 to satisfy the dependency, but, in so doing, unknowingly render the package or installation severely broken. A change [5] has been proposed as part of a prospective solution to this issue. It creates a new release of sushy from the change which fixes the first bug [2]. Review comments [6] discuss basing the new release on a more recent stable/train change to pick up other bug fixes and, less importantly, backward compatible feature modifications and enhancements which merged before the change from which 2.0.0 was created. Backward compatible feature modifications and enhancements are interspersed in time among the bug fixes. Once a new release is available, the sushy entry in driver-requirements.txt on stable/train would be updated. However, apparently, the stable branch policy prevents releases from being done at a point earlier than the last release within a given cycle [6], which was 2.0.0. Another possible resolution which comes to mind is to change the definition of the sushy dependency in driver-requirements.txt [1] from "sushy>=1.9.0" to "sushy>=2.0.0". Does anyone have a suggestion on how to proceed? Thank you, Rick [1] https://opendev.org/openstack/ironic/src/commit/b8ae681b37eec617736ac4a507e… [2] https://review.opendev.org/#/c/666253/ [3] https://review.opendev.org/#/c/668936/ [4] https://review.opendev.org/#/c/669889/ [5] https://review.opendev.org/#/c/688551/ [6] https://review.opendev.org/#/c/688551/1/deliverables/train/sushy.yaml@14

10 16

[magnum][kubernetes] Train release updates
by feilong 24 Oct '19

24 Oct '19

Hi all, Glad to see the Train just released. I'd like to take this opportunity to highlight some cool features Magnum team have done in this cycle. Please feel free to pop up at #openstack-containers for any feedback. Thank you. 1. Fedora CoreOS driver Finally we get this done in this cycle which is quite important and urgent. Because the Fedora Atomic will be end of life soon (at the end of this Nov?). Now with stable/train, user can create k8s cluster based on Fedora CoreOS image. We're welcome for any feedback about this. 2. Node group This is a quite large feature, thanks for the great work from CERN team and StackHPC. With this feature, user can create different node groups with different specs. 3. Rolling upgrade Now user can do rolling upgrade for both k8s version and the node operating system without downtime, it's a quite important feature. 4. Auto healing With auto healing feature, a small service will be deployed on k8s cluster to monitor the health status of the cluster and replace broken node when failure detected. 5. Volume based k8s nodes Now user can create a Kubernetes cluster with nodes based on volume and the volume type is configurable. User can even set the volume type of etcd volume which is very useful for cloud providers who want to use SSD for etcd and k8s nodes. 6. Private cluster As a security best practice, isolating Kubernetes clusters from Internet access is one of the most desired features for enterprise users. Now user has the flexibility to create a private cluster by default, only allow the API exposed on Internet or fully opened. -- Cheers & Best regards, Feilong Wang (王飞龙) ------------------------------------------------------ Senior Cloud Software Engineer Tel: +64-48032246 Email: flwang(a)catalyst.net.nz Catalyst IT Limited Level 6, Catalyst House, 150 Willis Street, Wellington ------------------------------------------------------

2 2

[ironic]: Timeout reached while waiting for callback for node
by fsbiz＠yahoo.com 24 Oct '19

24 Oct '19

Hi folks, We have a 300 strong ironic cluster which has been relatively stable for the most part.Occasionally we get the IPA timeout errors. | | 2019-10-20 23:33:37.994 | 2019-10-20 23:33:37.421 265272 ERROR nova.virt.ironic.driver [req-71933c39-2ac4-497b-8cdc-9578889f42fe cda0725e2a68445db857f56c49f77faf 53a4a4738d9f49e99fe580c1bd51f950 - default default] Error deploying instance c51368f6-b9d8-4c3c-bd63-d1f84ee26bfb on baremetal node 83dc67d4-36be-4004-8847-afd9b969de0e.: InstanceDeployFailure: Failed to provision instance c51368f6-b9d8-4c3c-bd63-d1f84ee26bfb: Timeout reached while waiting for callback for node 83dc67d4-36be-4004-8847-afd9b969de0e | Here's my troubleshooting so far. DHCP: All good. Oct 22 09:49:58 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPDISCOVER(ns-601d0738-39) 6c:b3:11:4f:87:acOct 22 09:49:58 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPOFFER(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:acOct 22 09:50:02 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPREQUEST(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:acOct 22 09:50:02 sc-control03 dnsmasq-dhcp[965699]: 3076961402 DHCPACK(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:ac host-10-33-23-94Oct 22 10:19:47 sc-control03 dnsmasq-dhcp[965699]: 0 DHCPRELEASE(ns-601d0738-39) 10.33.23.94 6c:b3:11:4f:87:acOct 22 10:21:23 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPDISCOVER(ns-601d0738-39) 6c:b3:11:4f:87:acOct 22 10:21:23 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPOFFER(ns-601d0738-39) 10.33.23.187 6c:b3:11:4f:87:acOct 22 10:21:26 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPREQUEST(ns-601d0738-39) 10.33.23.187 6c:b3:11:4f:87:acOct 22 10:21:26 sc-control03 dnsmasq-dhcp[965699]: 2188330678 DHCPACK(ns-601d0738-39) 10.33.23.187 6c:b3:11:4f:87:ac host-10-33-23-Oct 22 10:22:46 sc-control03 dnsmasq-dhcp[965699]: 3549042952 DHCPDISCOVER(ns-601d0738-39) 6c:b3:11:4f:87:ac Ironic conductor logs: (shows ironic state proceeding to Power ON, ironic state: deploying. Then 30 minutes later timeout since IPA failed to check-in 2019-10-22 09:49:03.746 267066 DEBUG ironic.drivers.modules.deploy_utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Deploy boot mode is uefi for d465fa17-a2ea-43ff-a3a7-033cbb497774. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:798 2019-10-22 09:49:03.747 267066 DEBUG ironic.common.pxe_utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Building PXE config for node d465fa17-a2ea-43ff-a3a7-033cbb497774 create_pxe_config /usr/lib/python2.7/site-packages/ironic/common/pxe_utils.py:222 2019-10-22 09:49:03.748 267066 DEBUG ironic.drivers.modules.deploy_utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Deploy boot mode is uefi for d465fa17-a2ea-43ff-a3a7-033cbb497774. get_boot_mode_for_deploy /usr/lib/python2.7/site-packages/ironic/drivers/modules/deploy_utils.py:798 2019-10-22 09:49:04.516 267066 DEBUG ironic.drivers.modules.pxe [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Fetching necessary kernel and ramdisk for node d465fa17-a2ea-43ff-a3a7-033cbb497774 _cache_ramdisk_kernel /usr/lib/python2.7/site-packages/ironic/drivers/modules/pxe.py:382 2019-10-22 09:49:17.761 267066 INFO ironic.conductor.utils [req-fabea5a4-da6c-48be-834a-49763c23b0e9 1ded19959aa94e53b6487d7c4f8d3371 acf8cd411e5e4751a61d1ed54e8e874d - default default] Successfully set node d465fa17-a2ea-43ff-a3a7-033cbb497774 power state to power on by rebooting. ===== 30 minutes later timeout. IPA failed to check-in 2019-10-22 10:19:37.448 267066 INFO ironic.conductor.task_manager [req-762d8acf-0a55-4d8d-99d8-93020a14547c - - - - -] Node d465fa17-a2ea-43ff-a3a7-033cbb497774 moved to provision state "deploy failed" from state "wait call-back"; target provision state is "active" 2019-10-22 10:19:37.471 267066 DEBUG ironic.common.pxe_utils [req-762d8acf-0a55-4d8d-99d8-93020a14547c - - - - -] Cleaning up PXE config for node d465fa17-a2ea-43ff-a3a7-033cbb497774 clean_up_pxe_config /usr/lib/python2.7/site-packages/ironic/common/pxe_utils.py:288 TFTP logs: shows TFTP client timed out (weird). Any pointers here?tftpd shows ramdisk_deployed completed. Then, it reports that the client timed out. Oct 22 09:50:15 sc-ironic04 in.tftpd[318194]: RRQ from 10.33.23.94 filename d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk remapped to /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk Oct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 finished d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdiskOct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 timed out Oct 22 09:50:03 sc-ironic04 in.tftpd[318148]: remap: input: bootx64.efiOct 22 09:50:03 sc-ironic04 in.tftpd[318148]: remap: rule 3: rewrite: /tftpboot/bootx64.efiOct 22 09:50:03 sc-ironic04 in.tftpd[318148]: remap: rule 3: exitOct 22 09:50:03 sc-ironic04 in.tftpd[318148]: RRQ from 10.33.23.94 filename bootx64.efi remapped to /tftpboot/bootx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318148]: Error code 8: User aborted the transferOct 22 09:50:04 sc-ironic04 in.tftpd[318150]: remap: input: bootx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318150]: remap: rule 3: rewrite: /tftpboot/bootx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318150]: remap: rule 3: exitOct 22 09:50:04 sc-ironic04 in.tftpd[318150]: RRQ from 10.33.23.94 filename bootx64.efi remapped to /tftpboot/bootx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318150]: Client 10.33.23.94 finished bootx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318151]: remap: input: grubx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318151]: remap: rule 3: rewrite: /tftpboot/grubx64.efiOct 22 09:50:04 sc-ironic04 in.tftpd[318151]: remap: rule 3: exitOct 22 09:50:04 sc-ironic04 in.tftpd[318151]: RRQ from 10.33.23.94 filename grubx64.efi remapped to /tftpboot/grubx64.efiOct 22 09:50:05 sc-ironic04 in.tftpd[318151]: Client 10.33.23.94 finished grubx64.efiOct 22 09:50:05 sc-ironic04 in.tftpd[318152]: remap: input: /grub.cfg-01-6c-b3-11-4f-87-acOct 22 09:50:05 sc-ironic04 in.tftpd[318152]: remap: rule 2: rewrite: /tftpboot//grub.cfg-01-6c-b3-11-4f-87-acOct 22 09:50:05 sc-ironic04 in.tftpd[318152]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318152]: RRQ from 10.33.23.94 filename /grub.cfg-01-6c-b3-11-4f-87-ac remapped to /tftpboot//grub.cfg-01-6c-b3-11-4f-87-acOct 22 09:50:05 sc-ironic04 in.tftpd[318152]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-01-6c-b3-11-4f-87-acOct 22 09:50:05 sc-ironic04 in.tftpd[318152]: sending NAK (1, File not found) to 10.33.23.94Oct 22 09:50:05 sc-ironic04 in.tftpd[318153]: remap: input: /grub.cfg-0A21175EOct 22 09:50:05 sc-ironic04 in.tftpd[318153]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A21175EOct 22 09:50:05 sc-ironic04 in.tftpd[318153]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318153]: RRQ from 10.33.23.94 filename /grub.cfg-0A21175E remapped to /tftpboot//grub.cfg-0A21175EOct 22 09:50:05 sc-ironic04 in.tftpd[318153]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A21175EOct 22 09:50:05 sc-ironic04 in.tftpd[318153]: sending NAK (1, File not found) to 10.33.23.94Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: remap: input: /grub.cfg-0A21175Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A21175Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318154]: RRQ from 10.33.23.94 filename /grub.cfg-0A21175 remapped to /tftpboot//grub.cfg-0A21175Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A21175Oct 22 09:50:05 sc-ironic04 in.tftpd[318154]: sending NAK (1, File not found) to 10.33.23.94Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: remap: input: /grub.cfg-0A2117Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A2117Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318155]: RRQ from 10.33.23.94 filename /grub.cfg-0A2117 remapped to /tftpboot//grub.cfg-0A2117Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A2117Oct 22 09:50:05 sc-ironic04 in.tftpd[318155]: sending NAK (1, File not found) to 10.33.23.94Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: remap: input: /grub.cfg-0A211Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A211Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318156]: RRQ from 10.33.23.94 filename /grub.cfg-0A211 remapped to /tftpboot//grub.cfg-0A211Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A211Oct 22 09:50:05 sc-ironic04 in.tftpd[318156]: sending NAK (1, File not found) to 10.33.23.94Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: remap: input: /grub.cfg-0A21Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A21Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318157]: RRQ from 10.33.23.94 filename /grub.cfg-0A21 remapped to /tftpboot//grub.cfg-0A21Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: Client 10.33.23.94 File not found /tftpboot//grub.cfg-0A21Oct 22 09:50:05 sc-ironic04 in.tftpd[318157]: sending NAK (1, File not found) to 10.33.23.94Oct 22 09:50:05 sc-ironic04 in.tftpd[318158]: remap: input: /grub.cfg-0A2Oct 22 09:50:05 sc-ironic04 in.tftpd[318158]: remap: rule 2: rewrite: /tftpboot//grub.cfg-0A2Oct 22 09:50:05 sc-ironic04 in.tftpd[318158]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318178]: RRQ from 10.33.23.94 filename /EFI/centos/grub.cfg remapped to /tftpboot//EFI/centos/grub.cfgOct 22 09:50:05 sc-ironic04 in.tftpd[318178]: Client 10.33.23.94 finished /EFI/centos/grub.cfgOct 22 09:50:05 sc-ironic04 in.tftpd[318179]: remap: input: /EFI/centos/grub.cfgOct 22 09:50:05 sc-ironic04 in.tftpd[318179]: remap: rule 2: rewrite: /tftpboot//EFI/centos/grub.cfgOct 22 09:50:05 sc-ironic04 in.tftpd[318179]: remap: rule 2: exitOct 22 09:50:05 sc-ironic04 in.tftpd[318179]: RRQ from 10.33.23.94 filename /EFI/centos/grub.cfg remapped to /tftpboot//EFI/centos/grub.cfgOct 22 09:50:05 sc-ironic04 in.tftpd[318179]: Client 10.33.23.94 finished /EFI/centos/grub.cfgOct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.298 267066 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._sync_local_state' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:639Oct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.298 267066 INFO ironic.conductor.manager [-] Executing sync_local_stateOct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.305 267066 DEBUG ironic.common.hash_ring [-] Rebuilding cached hash rings ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:51Oct 22 09:50:07 sc-ironic04 ironic-conductor: 2019-10-22 09:50:07.466 267066 DEBUG ironic.common.hash_ring [-] Finished rebuilding hash rings, available drivers are ipmi, fake, snmp, fake_pxe, pxe_snmp, agent_ipmitool, pxe_ipmitool ring /usr/lib/python2.7/site-packages/ironic/common/hash_ring.py:56Oct 22 09:50:10 sc-ironic04 in.tftpd[318184]: remap: input: /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318184]: remap: rule 0: rewrite: /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318184]: remap: rule 0: exitOct 22 09:50:10 sc-ironic04 in.tftpd[318184]: RRQ from 10.33.23.94 filename /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318184]: Client 10.33.23.94 finished /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318185]: remap: input: /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318185]: remap: rule 0: rewrite: /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318185]: remap: rule 0: exitOct 22 09:50:10 sc-ironic04 in.tftpd[318185]: RRQ from 10.33.23.94 filename /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318185]: Client 10.33.23.94 finished /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318186]: remap: input: /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318186]: remap: rule 0: rewrite: /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318186]: remap: rule 0: exitOct 22 09:50:10 sc-ironic04 in.tftpd[318186]: RRQ from 10.33.23.94 filename /tftpboot/10.33.23.94.confOct 22 09:50:10 sc-ironic04 in.tftpd[318186]: Client 10.33.23.94 finished /tftpboot/10.33.23.94.confOct 22 09:50:15 sc-ironic04 in.tftpd[318192]: remap: input: d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernelOct 22 09:50:15 sc-ironic04 in.tftpd[318192]: remap: rule 3: rewrite: /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernelOct 22 09:50:15 sc-ironic04 in.tftpd[318192]: remap: rule 3: exitOct 22 09:50:15 sc-ironic04 in.tftpd[318192]: RRQ from 10.33.23.94 filename d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernel remapped to /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernelOct 22 09:50:15 sc-ironic04 ironic-conductor: 2019-10-22 09:50:15.898 267066 DEBUG futurist.periodics [-] Submitting periodic callback 'ironic.conductor.manager.ConductorManager._check_rescuewait_timeouts' _process_scheduled /usr/lib/python2.7/site-packages/futurist/periodics.py:639Oct 22 09:50:15 sc-ironic04 in.tftpd[318192]: Client 10.33.23.94 finished d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_kernelOct 22 09:50:15 sc-ironic04 in.tftpd[318194]: remap: input: d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdiskOct 22 09:50:15 sc-ironic04 in.tftpd[318194]: remap: rule 3: rewrite: /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdiskOct 22 09:50:15 sc-ironic04 in.tftpd[318194]: remap: rule 3: exitOct 22 09:50:15 sc-ironic04 in.tftpd[318194]: RRQ from 10.33.23.94 filename d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk remapped to /tftpboot/d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdisk Oct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 finished d465fa17-a2ea-43ff-a3a7-033cbb497774/deploy_ramdiskOct 22 09:50:44 sc-ironic04 in.tftpd[318194]: Client 10.33.23.94 timed out This has me stumped here. This exact failure seems to be happening 3 to 4 times a week on different nodes.Any pointers appreciated. thanks,Fred.

2 1

[ptg][neutron][all] Project onboarding session
by Slawek Kaplonski 23 Oct '19

23 Oct '19

Hi, As a Neutron team we are planning to do onboarding session on Wednesday morning during the Shanghai PTG. I’m wondering if there are any ways to reach to as much people as it is possible with information when this onboarding will take place and I’m sure that putting it just in PTG etherpad will not be enough for potential newcomers. So I have a question if there are any other ways to announce when exactly such onboarding session will be? And maybe how other teams are announcing that. Thx in advance for any tips on clues :) — Slawek Kaplonski Senior software engineer Red Hat

1 0

[nova] Which nova container service that nova/conf/compute.py map to
by yu.chengde＠99cloud.net 23 Oct '19

23 Oct '19

Hi, I have deployed a stein version openstack on server thought Kolla-ansible method. Then, I git clone the nova code, and ready to do coding in " nova/nova/conf/compute.py" However, many of nova containers include this file. So, I want to know that I should modify them all, or just pick a specific one. Thanks [root@chantyu kolla-ansible]# docker ps | grep nova 05f72e539974 kolla/centos-source-nova-compute:stein "dumb-init --single-…" 28 hours ago Up 2 hours nova_compute 7393a7d566ee kolla/centos-source-nova-libvirt:stein "dumb-init --single-…" 28 hours ago Up 5 hours nova_libvirt 9d8357cfa334 kolla/centos-source-nova-scheduler:stein "dumb-init --single-…" 32 hours ago Up 3 hours nova_scheduler 085b9da918df kolla/centos-source-nova-api:stein "dumb-init --single-…" 6 days ago Up 3 hours nova_api b80e9503e93e kolla/centos-source-nova-serialproxy:stein "dumb-init --single-…" 6 days ago Up 3 hours nova_serialproxy c15d41823a22 kolla/centos-source-nova-novncproxy:stein "dumb-init --single-…" 6 days ago Up 3 hours nova_novncproxy c30e47cd56c6 kolla/centos-source-nova-consoleauth:stein "dumb-init --single-…" 6 days ago Up 3 hours nova_consoleauth b7d5e9ba1f11 kolla/centos-source-nova-ssh:stein "dumb-init --single-…" 7 days ago Up 5 hours nova_ssh 3f81cd0a97ce kolla/centos-source-nova-conductor:stein "dumb-init --single-…" 7 days ago Up 3 hours nova_conductor [root@chantyu kolla-ansible]#

4 4

[tc][all] Community-wide goal Ussuri and V cycle forum collaboration idea
by Ghanshyam Mann 23 Oct '19

23 Oct '19

Hello Everyone, During the TC meeting on 10th Oct, we discussed the community-wide goal planning for Ussuri as well as the V cycle[1] but there was no consensus on V cycle planning so I am bringing this to ML for further thoughts. Ussuri cycle goal planning is all good here, That has been already started[2] and we have a dedicated forum session [3] for the same to discuss the goals in more detail. Question is for V cycle goal planning, whether we should discuss the V cycle goal in Ussuri goal fourm sessoin[3] or it is too early to kick off V cycle goal at least until we finalize U cycle goal first. I would like to list the below two options to proceed further (at least to decide if we need to change the existing U cycle goal forum sessions title). 1. Merge the Forum session for both cycle goal discussion (divide both in two half). This need forum session title and description change. 2. Keep forum session for U cycle goal only and start the V cycle over ML asynchronously. This will help to avoid any confusion or mixing the both cycle goal discussions. Thoughts? [1] http://eavesdrop.openstack.org/meetings/tc/2019/tc.2019-10-10-14.00.log.htm… [2] https://etherpad.openstack.org/p/PVG-u-series-goals [3] https://www.openstack.org/summit/shanghai-2019/summit-schedule/events/24398… -gmann

2 2

[Infra][Cinder] IBM Storage OpenStack CI Gerrit account issue
by Ling LL Chen 23 Oct '19

23 Oct '19

Hi Dear, This is Linda from IBM Storage cinder team. Currently we need to set up the OpenStack CI environment. But now we failed to configure the Gerrit server with ssh key pair. We only know the Gerrit account is "ibm_storage_ci", but we don't know which mail address used for this account. Would you please help to check this Gerrit account "ibm_storage_ci" and provide the detail information with us? Thanks. Regards, Chen Ling(Linda,陈玲) ------------------------------------------- Cloud Storage Solutions Development (CSSD) IBM System SRA troubleshooting wiki: https://apps.na.collabserv.com/wikis/home?lang=en-us#!/wiki/W9c4e08533b49_4… Tel: (8621)6092-8292 Ext:28292 Address: 3F, Building 10, 399 Keyuan Road, Zhangjiang Hi-Tech Park, Pudong New District, Shanghai 201203, China -------------------------------------------

2 1

[sigs][publiccloud][publiccloud-wg][publiccloud-sig] Bi-weekly meeting for the Public Cloud SIG tomorrow CANCELLED
by Tobias Rydberg 23 Oct '19

23 Oct '19

Hi all, Need to cancel tomorrows meeting since I will be on a flight at that time. Hope to see most of you in Shanghai in less than 2 weeks. Cheers, Tobias -- Tobias Rydberg Senior Developer Twitter & IRC: tobberydberg www.citynetwork.eu | www.citycloud.com INNOVATION THROUGH OPEN IT INFRASTRUCTURE ISO 9001, 14001, 27001, 27015 & 27018 CERTIFIED

1 0

[Openstack] OpenStack Train for Ubuntu 18.04 LTS
by Corey Bryant 23 Oct '19

23 Oct '19

The Ubuntu OpenStack team at Canonical is pleased to announce the general availability of OpenStack Train on Ubuntu 18.04 LTS via the Ubuntu Cloud Archive. Details of the Train release can be found at: https://www.openstack.org/software/train To get access to the Ubuntu Train packages: Ubuntu 18.04 LTS ----------------------- You can enable the Ubuntu Cloud Archive pocket for OpenStack Train on Ubuntu 18.04 installations by running the following commands: sudo add-apt-repository cloud-archive:train sudo apt update The Ubuntu Cloud Archive for Train includes updates for: aodh, barbican, ceilometer, ceph (14.2.2), cinder, designate, designate-dashboard, dpdk (18.11.2), glance, gnocchi, heat, heat-dashboard, horizon, ironic, keystone, libvirt (5.4.0), magnum, manila, manila-ui, mistral, murano, murano-dashboard, networking-arista, networking-bagpipe, networking-bgpvpn, networking-hyperv, networking-l2gw, networking-mlnx, networking-odl, networking-ovn, networking-sfc, neutron, neutron-dynamic-routing, neutron-fwaas, neutron-lbaas, neutron-lbaas-dashboard, neutron-vpnaas, nova, octavia, openstack-trove, openvswitch (2.12.0), panko, placement, qemu (4.0), sahara, sahara-dashboard, senlin, swift, trove-dashboard, vmware-nsx, watcher, and zaqar. For a full list of packages and versions, please refer to: http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/train_vers… Python support ------------------- The Train release of Ubuntu OpenStack is Python 3 only; all Python 2 packages have been dropped in Train. Branch package builds ----------------------------- If you would like to try out the latest updates to branches, we deliver continuously integrated packages on each upstream commit via the following PPA’s: sudo add-apt-repository ppa:openstack-ubuntu-testing/mitaka sudo add-apt-repository ppa:openstack-ubuntu-testing/ocata sudo add-apt-repository ppa:openstack-ubuntu-testing/queens sudo add-apt-repository ppa:openstack-ubuntu-testing/rocky sudo add-apt-repository ppa:openstack-ubuntu-testing/train Reporting bugs ------------------- If you have any issues please report bugs using the 'ubuntu-bug' tool to ensure that bugs get logged in the right place in Launchpad: sudo ubuntu-bug nova-conductor Thanks to everyone who has contributed to OpenStack Train, both upstream and downstream. Special thanks to the Puppet OpenStack modules team and the OpenStack Charms team for their continued early testing of the Ubuntu Cloud Archive, as well as the Ubuntu and Debian OpenStack teams for all of their contributions. Enjoy and see you in Ussuri! Corey (on behalf of the Ubuntu OpenStack team)

1 0

[RelMgmt] Release team meeting cancelled Oct 24
by Sean McGinnis 23 Oct '19

23 Oct '19

Hey all, We discussed this in last week's meeting, but just a reminder that we will be skipping the release team meeting this Thursday, October 24. We are reaching the final deadline for the stable/queens branch to transition to Extended Maintenance. Please watch for anything related to that and help review final release patches if you can. Anything else that comes up, we can just address ad hoc in the #openstack-release channel. Thanks! Sean

1 0

"Large Scale SIG" discussion in Shanghai
by Thierry Carrez 23 Oct '19

23 Oct '19

Hi everyone, As OpenStack clusters grow larger, they hit scaling limitations in a number of components. To work around this problem, operators create multiple clusters. In recent visits to OpenStack users, I noticed a common theme around people interested in working to scale single clusters past their current limits. In the past we had various groups trying to identify bottlenecks or work on alternate technologies (RabbitMQ replacements): the Performance WG, or the "Large deployment" WG. However those groups had a bit of a wide focus, and/or are not active anymore. In Shanghai we'll discuss creating a "Large Scale SIG" that would tackle specifically those single-cluster-scaling issues. The idea is that by sharing performance analysis, it can help identify key bottlenecks. By pooling development resources, it can push fixes or new features to address those issues. I directly reached out to several people and organizations interested in working on that. Please join the session in Shanghai if you'll be present there and are interested in discussing this topic: https://www.openstack.org/summit/shanghai-2019/summit-schedule/events/24405… -- Thierry Carrez (ttx)

1 0

[tacker][ptg] Ussuri PTG Planning for Tacker
by Dharmendra Kushwaha 23 Oct '19

23 Oct '19

Hello Everyone, I have created an etherpad [1] to collect the topics for Shanghai PTG discussion for Tacker. Please start drafting/adding your topics which you want to discuss in PTG. Even if anyone not be physically available for PTG, but want to discuss their topics, please add in etherpad, we will discuss. [1]: https://etherpad.openstack.org/p/Tacker-PTG-Ussuri Thanks & Regards Dharmendra Kushwaha

1 0

[tc][goal]: Ussuri community goal candidate 2: 'Switch remaining legacy jobs to Zuul v3 and drop legacy support'
by Ghanshyam Mann 23 Oct '19

23 Oct '19

Hello Everyone, As you might have seen on the ML about Ussuri community goal feedback for each proposed candidate [1], 2nd candidate is 'Switch remaining legacy jobs to Zuul v3 and drop legacy support' [2]. Andreas has briefed this goal in etherpad[2]. "Convert the remaining legacy jobs to Zuul v3 features. Grenade might be the largest open issue. This means especially that devstack-gate will not support Ussuri but only older releases. Having all jobs migrated to Zuul v3 native will allow easier maintenance and sharing between them. " Zuul has good documentation about migrating the legacy job to zuulv3 native. devstack and tempest have already migrated all their legacy jobs and so does many projects also ( if not all but few jobs for sure). Keeping legacy jobs are very costly not just in term of maintenance but also for doing the community-wide work. I have experienced the same during the migration of upstream CI from xenial - > Bionic in stein and IPV6 goal in Train. Currently, we do not have any volunteers to lead this goal. I would like to ask for the volunteer who can drive this goal do the pre-work about defining the goal and then be champion for this goal. Feel Free to reply to this email or ping me on #openstack-tc IRC channel. [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010287.… [2] https://etherpad.openstack.org/p/PVG-u-series-goals -gmann

2 2

[helm][loci][rpm][ironic] Please fill deployment tool capabilities
by Thierry Carrez 22 Oct '19

22 Oct '19

OpenStack-Helm, LOCI, RPM-packaging & Bifrost (ironic) folks: We recently started to display deployment tools capabilities over at: https://www.openstack.org/software/project-navigator/deployment-tools However OpenStack-Helm, LOCI, RPM Packaging and Bifrost display only the (default) component:keystone capability, because they were not filled when we we last asked[1][2]. As such, they look a bit sad. Please propose a patch to the following file: https://opendev.org/osf/openstack-map/src/branch/master/deployment_tools.ya… Capabilities are described in detail in: https://opendev.org/osf/openstack-map/src/branch/master/deployment_tools_ca… Let me know if you have any question, [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/007585.html [2] http://lists.openstack.org/pipermail/openstack-discuss/2019-July/008150.html -- Thierry Carrez (ttx)

2 1

[Watcher] No meeting on October 23 and November 6
by li.canwei2＠zte.com.cn 22 Oct '19

22 Oct '19

Hi all, I'm no time tomorrow and Nov 6 is during the Openstack Shanghai Summit, so canceling the IRC meeting. thanks! licanwei

1 0

[neutron] CI meeting cancelled
by Slawek Kaplonski 22 Oct '19

22 Oct '19

Hi, I’m on training this week and I can’t lead CI meeting. As also some other people who usually attends our CI meeting are also on the same training, lets cancel the meeting this week. We will back with CI meetings next week, so 29.10. — Slawek Kaplonski Senior software engineer Red Hat

1 0

[openstack-discuss][Airship] Call for PTG Participation for Airship
by prakash RAMCHANDRAN 22 Oct '19

22 Oct '19

Hi all, We seek active participation from community to enable Airship design and deployment for new release v2 in progress.Join us at PTG & mini-PTG to complete the Specs for MVP Beta (v2) before we get to November monthly sprint targets.Add your name topics and any topics you want to be bring forth. PTG at Shnnghai - Airship PTG Ussuri Half day Thursday Nov 7th, full day Friday Nov 8th https://etherpad.openstack.org/p/airship-ptg-ussuri Airship KubeCon mini-PTG San DiegoMonday, 12:30-6:30 [planning on kicking off @ 1pm] https://etherpad.openstack.org/p/airship-kubecon-san-diego ThanksPrakash / For Airship PTG/mini-PTGs

1 0

RMQ notifications.info has no consumers
by Albert Braden 21 Oct '19

21 Oct '19

I installed the RMQ management plugin in my dev and qa clusters and noticed a queue called notifications.info that has 0 consumers: root@us01odc-qa-ctrl1:~# rabbitmqctl list_queues|grep notific notifications.info 15496 If I cold-start RMQ it starts without a notifications.info queue but after I create a VM it appears with 1 or 2 messages and then gradually grows over time. The messages in it appear to be Openstack operations: {"oslo.message": "{\"_context_domain\": null, \"_context_roles\": [\"admin\"], \"_context_global_request_id\": null, \"event_type\": \"dns.domain.exists\", \"_context_edit_managed_records\": false, \"_context_service_catalog\": null, \"timestamp\": \"2019-10-10 21:02:01.524052\", \"_context_all_tenants\": true, \"_unique_id\": \"326b8e30d10f40279d806384e7023954\", \"_context_resource_uuid\": null, \"_context_request_id\": \"req-a16c0cd5-94c3-4bf7-90ed-3ab677cc8e66\", \"_context_is_admin_project\": true, \"_context_tsigkey_id\": null, \"_context_client_addr\": null, \"payload\": {\"updated_at\": \"2019-09-24T15:08:55.000000\", \"minimum\": 3600, \"ttl\": 60, \"serial\": 1569337730, \"deleted_at\": null, \"id\": \"67840d5f-6b98-449c-80c4-4d3e3ee74d19\", \"parent_zone_id\": null, \"retry\": 600, \"transferred_at\": null, \"version\": 470, \"type\": \"PRIMARY\", \"email\": \"linux-admin-core(a)synopsys.com\", \"status\": \"ACTIVE\", \"description\": null, \"deleted\": \"0\", \"shard\": 1656, \"action\": \"NONE\", \"expire\": 86400, \"audit_period_beginning\": \"2019-10-10T20:02:01.486946\", \"masters\": [], \"name\": \"sg.us01-qa.synopsys.com.\", \"tenant_id\": \"80a60cda79be4ec48e12683478f0cf3b\", \"created_at\": \"2019-09-05T17:18:47.000000\", \"pool_id\": \"794ccc2c-d751-44fe-b57f-8894c9f5c842\", \"refresh\": 3545, \"delayed_notify\": false, \"audit_period_ending\": \"2019-10-10T21:02:01.486946\", \"attributes\": []}, \"_context_auth_token\": null, \"_context_system_scope\": null, \"_context_original_tenant\": null, \"_context_user_identity\": \"- - - - -\", \"_context_show_deleted\": false, \"_context_tenant\": null, \"_context_hide_counts\": false, \"priority\": \"INFO\", \"_context_read_only\": false, \"_context_is_admin\": true, \"_context_abandon\": null, \"_context_project_domain\": null, \"_context_user\": null, \"_context_user_domain\": null, \"publisher_id\": \"producer.us01odc-qa-ctrl2\", \"message_id\": \"0ceae8e3-e6df-4b08-8b1e-c6d8e75d456d\", \"_context_project\": null}", "oslo.version": "2.0"} I didn't install the management plugin in prod, but it appears that I have the same issue there: root@us01odc-p01-ctrl1:~# rabbitmqctl list_queues|grep notific notifications.info 226869 What should be consuming the notifications.info queue?

4 3