September 2021 - openstack-discuss

Not registering Hypervisors
by Karera Tony 27 Sep '21

27 Sep '21

Hello Team, I have deployed Openstack Victoria using Kolla-ansible on Ubuntu 20.04 and ceph as the backend storage for Nova, Cinder and Glance. It finished with no error but it has failed to register any on the Compute Nodes under Hypervisors. Any idea on how to resolve this ? Regards Tony Karera

5 22

[neutron] upgrading port binding:profile.allocation value for port-resource-request-groups API extension
by Balazs Gibizer 27 Sep '21

27 Sep '21

Hi Neutrinos! We found a technical challenge during implementing the port-resource-request-groups API extension[1]. That extension changes the format of the port.resoruce_request as well as the format of the port.binding:profile.allocation. The former is a calculated field on the port so that is easy. However the bindig:profile is persisted in the database so data migration is needed. What is the canonical way to do such DB data translation in Neutron? Can we translate the data in place during alembic migration? Or should we do some kind of online data migration when the data is translated by neutron when it is read from the db? cheers, gibi [1] https://review.opendev.org/c/openstack/neutron/+/805637/5

2 6

Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 27 Sep '21

27 Sep '21

Hi, after some time of using newly deployed kolla-ansible I have fluentd errors: on all controllers, storage and network nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” on compute nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” and on monitoring node: 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:56:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:56:32.140741991 +0200 chunk="5ccf89e1feb471b7ce25a26d37977baa" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:56:31 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:58:03 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=90.91785193886608 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:58:03 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89e1feb471b7ce25a26d37977baa" 2021-09-27 15:00:01 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/server.log; waiting 5 seconds 2021-09-27 15:00:01 +0200 [info]: #0 following tail of /var/log/kolla/kafka/server.log 2021-09-27 15:00:10 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=127.54797655600123 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:02:29 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=137.74220423400402 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=38.81670297612436 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:13 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/controller.log; waiting 5 seconds 2021-09-27 15:03:13 +0200 [info]: #0 following tail of /var/log/kolla/kafka/controller.log What seems to be the problem? Is there any way to run more than one fluentd worker in kolla? Best regards Adam Tomas

1 0

[keystone] Correct policies setup for System Administrators
by Jan Wasilewski 27 Sep '21

27 Sep '21

Hello, I am preparing policies configuration before an upgrade to the newer OpenStack release(Stein) and I would like to create a group of System Administrators to be able to get i.e. a list of all projects in the OpenStack cloud. I was following a description from this page [1] but it seems my admin user is able to get only a list of projects where it is directly added(i.e. with member role, reader role, or admin role). I am just wondering if we can list all of the OpenStack projects by System Administrator user without role reader added to every single project? To summarize what steps were done so far: - Original policy.json file which was used is here [2] - Only one option was changed so far: from: "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id", to: "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)", - Output for command: openstack role assignment list --system all --role member --role reader +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | e39e97c23bfe45d1a2f9689b6985f990 | | a0841b83f583477887219f27dd95477b | | | all | False | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+Shows only role reader, not role member, which is a bit strange if we compare with linked page above. But we have this in implied roles:openstack implied role list +----------------------------------+-----------------+----------------------------------+-------------------+ | Prior Role ID | Prior Role Name | Implied Role ID | Implied Role Name | +----------------------------------+-----------------+----------------------------------+-------------------+ | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 3f20cb7be46346a8b2ba65c4684d50a3 | member | | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | | 3f20cb7be46346a8b2ba65c4684d50a3 | member | e39e97c23bfe45d1a2f9689b6985f990 | reader | +----------------------------------+-----------------+----------------------------------+-------------------+- Admin roles are grouped in a group ATM.Admin: openstack role assignment list --names --system all --role admin: +-------+---------------------+-------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+---------------------+-------------------+---------+--------+--------+-----------+ | admin | | ATM.Admin@Default | | | all | False | | admin | admin@Default | | | | all | False | | admin | jwasilewski@Default | | | | all | False | +-------+---------------------+-------------------+---------+--------+--------+-----------+Just to be sure that IDs are linked, we can check it here:openstack role assignment list --system all --role admin +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | a3c7bb5d06884b048c1bfb4403b82b42 | | a0841b83f583477887219f27dd95477b | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | 19416fe5a2da45c88eb66c3aaf856c73 | | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | f42df418fd404d04b9bdabf2f1b49fd9 | | | | all | False | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ So by linking roles(implied roles): admin(a3c7bb5d06884b048c1bfb4403b82b42 ) -> member(3f20cb7be46346a8b2ba65c4684d50a3) -> reader(e39e97c23bfe45d1a2f9689b6985f990). Correlation is visible, based on that my user(jwasilewski) should retrieve a full project list, but it seems only three projects are visible where this user is an admin. I do not want to add my user as a reader to every single project to be able to list all of them. Is there a way how to make it or the only way is to add this role(reader) for a user to all projects? Thank you in advance for any suggestions. Best regards, Jan Wasilewski [1] https://docs.openstack.org/keystone/stein/admin/service-api-protection.html… -> https://docs.openstack.org/keystone/latest/admin/service-api-protection.htm… [2] https://paste.openstack.org/show/bq0HgyqouZF1vywKVkGn/

1 0

[neutron] CI update
by Slawek Kaplonski 27 Sep '21

27 Sep '21

Hi, Due to very urgent things which I had yesterday, I wasn't able to run Neutron CI meeting as usually. Fortunatelly we don't have many new issues in our CI. There is only one new issue in our scenario jobs which I wanted to discuss [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI as well. See [2] or [3] for example. I'm not sure about Ironic jobs but in Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. [1] https://bugs.launchpad.net/neutron/+bug/1944201 [2] https:// e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ logs/screen-q-agt.txt [3] https://storage.bhs.cloud.ovh.net/v1/ AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ neutron-ovs-tempest-slow/88f8bb7/job-output.txt -- Slawek Kaplonski Principal Software Engineer Red Hat

5 8

[neutron] Bug deputy report (week starting on 2021-09-20)
by Bernard Cafarelli 27 Sep '21

27 Sep '21

Hello neutrinos, here is the list of bugs reported for last week, overall all bugs have assignees or fixes in progress. Both critical bugs are merged and with pending xena backports (this week is the final RCs week) In details: Critical * neutron-openvswitch-agent crashes on start with firewall config of br-int - https://bugs.launchpad.net/neutron/+bug/1944201 First seen in Ironic jobs but we also see it in our jobs, fix by ralonsoh merged: https://review.opendev.org/c/openstack/neutron/+/810592 (pending xena backport) * neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails with Port already has an attached device - https://bugs.launchpad.net/tripleo/+bug/1943708 Previously reported as https://bugs.launchpad.net/neutron/+bug/1943679, fix with revert proposed by slaweq and merged: https://review.opendev.org/c/openstack/neutron/+/809550 (pending xena backport) High * Traffic leaked from dhcp port before vlan tag is applied - https://bugs.launchpad.net/neutron/+bug/1930414 Possible security issue that was under investigation and now public, assigned to rubasov * OVN does not update the network segment tag - https://bugs.launchpad.net/neutron/+bug/1944708 Assigned to lucasagomes, fix in review: https://review.opendev.org/c/openstack/neutron/+/810704 Medium * Fix OVN metadata agent functional tests for Chassis_Private - https://bugs.launchpad.net/neutron/+bug/1944460 Functional OVN tests failing when using newer OVN, fix merged: https://review.opendev.org/c/openstack/neutron/+/810309 * IPv6 slaac subnet creation causes FixedIpsSubnetsNotOnSameSegment error - https://bugs.launchpad.net/neutron/+bug/1944948 Error raised when creating an IPv6 slaac subnet with multisegment, assigned to elvira Incomplete * Instances with SRIOV ports loose access after failed live migrations - https://bugs.launchpad.net/neutron/+bug/1944619 SRIOV live migration issue, liuyulong gave some pointers -- Bernard Cafarelli

1 0

[release] Release countdown for week R-1, Sep 27 - Oct 1
by Herve Beraud 27 Sep '21

27 Sep '21

Development Focus ----------------- We are on the final mile of the Xena development cycle! Remember that the Xena final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 30 is the deadline for final Xena release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 6, 2021. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the Yoga development cycle, including discussing sessions content in preparation of the PTG on the week of October 18. Actions ------- Watch for any translation patches coming through on the stable/xena branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/xena branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final Xena release: October 6 Yoga PTG: October 18-22 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[neutron][ovn] support for stateless NAT for floating ip in ml2 ovn
by Moshe Levi 27 Sep '21

27 Sep '21

Hi all, OVN support stateless NAT operations [1] for use case of 1:1 mapped between inner and external ips, i.e dnat_and_snat rule. In openstack is the floating ip use-case. Looking on ml2 ovn support it seem that it only support floating ip with connection tracking. Can ml2 ovn support also the stateless NAT option? Is there concerns using stateless NAT? [1] - https://patchwork.ozlabs.org/project/openvswitch/cover/1570154179-14525-1-g…

3 8

[requirements] Please consider bumping docker to 5.0.2 in Xena u-c
by Pierre Riteau 27 Sep '21

27 Sep '21

Hello, Xena upper constraints currently contain docker 5.0.1, released on August 31. This version includes a critical issue [1] which was fixed by version 5.0.2 released the next day. Except for a docstring update this is the only change, according to git diff. Would it be possible to update Xena u-c to use docker 5.0.2? We currently have to work around it in Kayobe to avoid installing the buggy version [2]. Thanks, Pierre Riteau (priteau) [1] https://github.com/docker/docker-py/issues/2885 [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-targ…

3 2

[zun][requirements] request to blacklist docker 5.0.1 for X release
by Hongbin Lu 26 Sep '21

26 Sep '21

Hi, It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. [1] https://github.com/docker/docker-py/issues/2885 [2] https://bugs.launchpad.net/zun/+bug/1945088 Best regards, Hongbin

2 2

anyone running ovn with ovs+dpdk
by Satish Patel 26 Sep '21

26 Sep '21

Folks, Anyone running ovn with dpdk on ubuntu 20.04 (Focal) with ovs version 2.13.3 ? I am trying to do that but my ovs switch keep crashing as soon as i spin up vm and it creates socket file in /var/run/openvswitch and I am not able to understand what is wrong with it. (no generating crash dump also) Please advise or share your thoughts.

1 0

All Devstack Based Jobs Failing to Create Swap on master
by Clark Boylan 24 Sep '21

24 Sep '21

Hello everyone, At approximately 12:53 UTC September 24 (today) [0] Devstack's Xena branch was created. Due to an unexpected side effect in Devstack's Zuul configs this has caused all Devstack based jobs against master and possible stable/xena to RETRY_LIMIT and fail. The underlying issue you'll see in the jobs is that mkswap cannot create swap because it already exists [1]. This happens because Devstack's pre-run playbook is running twice [2] and the configure-swap role is not idempotent. The role is running twice because the Devstack base job is loading master and stable/xena's configs and composing them together which results in two pre-run playbooks being selected one for each branch. This happens because Devstack's stable/xena branch explicitly states that its configs apply to master and feature/r1 branches [3]. I've pushed up a change [4] to remove this Zuul configuration from the Xena branch which I expect will fix things for us. [0] https://zuul.opendev.org/t/openstack/build/ebe77aee0c104c9e9b917ff28a4731ee… [1] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [2] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [3] https://opendev.org/openstack/devstack/src/branch/stable/xena/.zuul.yaml#L1… [4] https://review.opendev.org/c/openstack/devstack/+/810947

2 2

[all][tc] What's happening in Technical Committee: summary 24th Sept, 21: Reading: 5 min
by Ghanshyam Mann 24 Sep '21

24 Sep '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 23rd Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-23-15.00.log.html * We will have next week's meeting on Sept 30th, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 29th. 2. What we completed this week: ========================= * Moved Cloud Research SIG to active state[2] 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ * TC is using the etherpad[3] for Xena cycle working item. We will be checking and updating the status biweekly on the same etherpad. * Current status is: 7 completed, 6 in-progress Open Reviews ----------------- * Five open reviews for ongoing activities[4]. Add project health check tool ----------------------------------- * We are automating the project health checks. Currently, every TC member is assigned a number of projects to keep checking the project health/ listening to their issues but this is not so successful so far. * Rico has proposed the code[5] to collect the contribution stats and we are in the process of documenting the usage and interpretation of those stats. Stable Core team process change --------------------------------------- * As discussed in Shanghai and Xena PTG, there are some concerns over the current stable core team process (especially now when there are very few global stable team members) and to improve the process, mnaser has proposed the draft resolution[6] . Feel free to provide early feedback if you have any. Removing stable release 'Unmaintained' phase ------------------------------------------------------- * 'Unmaintained' phase in the stable release process has been confusing than adding any value. * Elod has proposed the removal of the 'unmaintained' phase[7], feel free to reply if any objection otherwise we will merge the patch on Monday(27th Sept). Places for projects to spreading the word ------------------------------------------------- * Amy has started to list down all the places that projects can use to spread the word out about your project which will help to get additional contributors and users[8]. She also sent emails to PTLs. * Please make note of it and use such places/blogs to highlights/educate operators/users/community etc. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[9] or what all tags can be cleaned up. * We are still waiting for the operator's response to the email on openstack-disscuss ML[10]. If you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://review.opendev.org/c/openstack/governance-sigs/+/809887 [3] https://etherpad.opendev.org/p/tc-xena-tracker [4] https://review.opendev.org/q/projects:openstack/governance+status:open [5] https://review.opendev.org/c/openstack/governance/+/810037 [6] https://review.opendev.org/c/openstack/governance/+/810721 [7] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02501… [8] https://review.opendev.org/c/openstack/project-team-guide/+/810766 [9] https://governance.openstack.org/tc/reference/tags/index.html [10] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02480… [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[ops][neutron] Is it possible to "lock" a floating IP to an instance ?
by Massimo Sgaravatto 24 Sep '21

24 Sep '21

Hello I have the following use case: A user creates a VM and associates a floating IP to such instance Is in some way possible to prevent that the floating IP is disassociated from that instance by another user of the same project ? If it helps, the user owning the instance could be admin (but allowing only the admin user to manage floating IPs is not an option) Thanks, Massimo

2 2

[Sahara]Currently about the development of the Sahara community there are some points
by Juntingqiu Qiujunting (邱军婷) 24 Sep '21

24 Sep '21

Hi all: Currently about the development of the Sahara community there are some points as following: 1. About the schedule of the regular meeting of the Sahara project? What is your suggestion? How about the regular meeting time every Wednesday afternoon 15:00 to 16:30? 2. Regarding the Sahara project maintenance switch from StoryBoard to launchpad. https://storyboard.openstack.org/ https://blueprints.launchpad.net/openstack/ The reasons are as follows: 1. OpenSatck core projects are maintained on launchpad, such as nova, cinder, neutron, etc. 2. Most OpenStack contributors are used to working on launchpad. 3. Do you have any suggestions? If you think this is feasible, I will post this content in the Sahara community later. Thank you for your help. Thank you Fossen. --------------------------------- Fossen Qiu | 邱军婷 CBRD | 云计算与大数据研发部 T: 18249256272 E: qiujunting(a)inspur.com<mailto:qiujunting@inspur.com> 浪潮云海

3 2

Neutron FIP Qos with OVN Backend
by Ammad Syed 24 Sep '21

24 Sep '21

Hello, I am using Neutron wallaby with OVN backend. I am testing QoS FIP and it works fine. I have created a QoS policy and attached it with FIP, It works fine. Is it possible to set one of the QoS policy as default that each time I acquire a floating IP, the QoS should be attached to it by default ? Ammad

4 4

[devstack][requirements][stable/ussuri] New setuptools doesn't like suds-jurko?
by Neil Jerram 24 Sep '21

24 Sep '21

I use the stable/ussuri branches of devstack and tempest and have recently started seeing this stack.sh failure: Collecting suds-jurko===0.6 Downloading suds-jurko-0.6.tar.bz2 (143 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3.6 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"'; __file__='"'"'/tmp/pip-install-wsiug8gg/suds-jurko/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-h8s2x8w4 cwd: /tmp/pip-install-wsiug8gg/suds-jurko/ Complete output (1 lines): error in suds-jurko setup command: use_2to3 is invalid. IIUC that last error is from setuptools, specifically version 58.0.4. In previous runs with setuptools 57.4.0, this error did not occur. Does it mean that stable/ussuri global-requirements.txt for setuptools should have "<58" ? I haven't investigated why suds-jurko is pulled in, but I guess it's from the reference Neutron code - as opposed to something that my particular testing has added - and so this could affect anyone using stable/ussuri devstack. Best wishes, Neil -- Neil Jerram Principal Software Engineer Tigera neil(a)tigera.io | @neiljerram <http://twitter.com/neiljerram> Follow Tigera: Blog <https://blog.tigera.io/> | Twitter <https://twitter.com/tigeraio> | LinkedIn <https://www.linkedin.com/company/tigera/> Leader in Kubernetes Security and Observability <https://tigera.io/>

2 2

[release][qa][requirements][i18n] time to finalize stable branch setup
by Előd Illés 24 Sep '21

24 Sep '21

Hi, All the projects that are enabled in devstack by default have been branched, so it's time to - [requirements]: create branch for 'requirements' to be able to lift the requirements freeze - [qa]: do the necessary QA preparation patches ( https://wiki.openstack.org/wiki/QA/releases <https://wiki.openstack.org/wiki/QA/releases> ) - [i18n]: update the translation tools for the new stable series So I ask the PTLs of these teams to do the above necessary preparation. Thanks, Előd

1 0

[ironic] Yoga PTG
by Iury Gregory 23 Sep '21

23 Sep '21

Hello Ironicers! We are approaching the time of the next PTG (it's in October but we need to make some decisions). Our yoga etherpad is available[1], feel free to add any topics we shall discuss. In the ethercalc[2] you can find the slots we already booked, we may add more slots for APAC if someone is willing to run the meeting and discussions. Don't forget to register! [3]. [1] https://etherpad.opendev.org/p/ironic-yoga-ptg [2] https://ethercalc.openstack.org/8tum5yl1bx43 [3] https://openinfra-ptg.eventbrite.com -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

1 1

[neutron] Drivers meeting - Friday 24.09.2021 - cancelled
by Lajos Katona 23 Sep '21

23 Sep '21

Hi Neutron Team, Due to lack of agenda, let's cancel tomorrow's drivers meeting. Let's concentrate on the ongoing release and see You on the meeting next week :-) Lajos Katona (lajoskatona)

1 0

Re: RabbitMQ annoying disconnections [kolla-ansible, rabbitmq]
by bkslash 23 Sep '21

23 Sep '21

Hi, I’ve tried to set heartbeat_in_pthread = true, but nothing has changed after kolla-ansible reconfigure - still got disconnections and heartbeat_in_pthread option was present inside configs on all nodes. But in the meantime I tried to alternate rabbit configuration and I realized that Kolla ignored /etc/kolla/config/rabbitmq.conf file (of course at first I checked permissions, etc.) So I tried to put heartbeat = 600 inside /etc/kolla/config/rabbitmq/rabbitmq.conf - this overwrote rabbitmq.conf totally (I mean Kolla didn’t just add heartbeat = 600 to rabbitmq.conf on each controller node, but put ONLY this line in all rabbitmq.conf files ! - rest of the file was cleared). So finally I tried to create separate config for each controller node. So I copied default content of /etc/kolla/rabbitmq/rabbitmq.conf on each controller (because of course these files differs from each other - they have different IPs inside) to /etc/kolla/config/rabbitmq/controller1(2,3)/rabbitmq.conf And finally heartbeat = 600 started to work - no more disconnections. I left Oslo heartbeats and timeouts intact (with the values I’ve set previously) and I don’t have disconnections from services side too. So there are two questions: 1. Why heartbeat_in_pthread didn’t change situation? And if it’s deprecated it was kind of risky to use it long term (after upgrade to wallaby it would stop working anyway?) 2. Why Kolla-ansible ignored rabbitmq.conf file if it is located in /etc/kolla/config, and overwrites ALL content of rabbitmq.conf file when it is located in /etc/kolla/config/rabbitmq/ ? Best regards Adam Tomas > Wiadomość napisana przez Herve Beraud <hberaud(a)redhat.com <mailto:hberaud@redhat.com>> w dniu 21.09.2021, o godz. 08:12: > > > > Le sam. 18 sept. 2021 à 16:12, bkslash <bkslash(a)poczta.onet.pl <mailto:bkslash@poczta.onet.pl>> a écrit : > Hi Ben, > Thank you for the answer. I’m using open stack Victoria deployed with Kolla-ansible. > So I assume I should add (in /etc/kolla/config/global.conf) > > [oslo_messaging_rabbit] > heartbeat_in_pthread = True > > > But for Victoria (nova sample configuration file) it seems to be deprecated? > # DEPRECATED: Run the health check heartbeat thread through a native python > # thread by default. If this option is equal to False then the health check > # heartbeat will inherit the execution model from the parent process. For > # example if the parent process has monkey patched the stdlib by using > # eventlet/greenlet then the heartbeat will be run through a green thread > # (boolean value) > # This option is deprecated for removal. > # Its value may be silently ignored in the future. > #heartbeat_in_pthread = true > So will it work? > > > Yes it will work. > > Few months ago I proposed to remove this deprecation and to keep the option available [1], unfortunately my removal isn't yet merged. > Only the default value is changed and the option will be turned on by default. > > [1] https://review.opendev.org/c/openstack/oslo.messaging/+/800621 <https://review.opendev.org/c/openstack/oslo.messaging/+/800621> > > Best regards > Adam Tomas > > >> Wiadomość napisana przez Ben Nemec <openstack(a)nemebean.com <mailto:openstack@nemebean.com>> w dniu 17.09.2021, o godz. 18:57: >> >> This sounds like you're hitting the heartbeat/eventlet bug. Basically eventlet won't schedule the heartbeat thread unless something is regularly waking up the api threads. Depending on what release you're running, you may be able to set heartbeat_in_pthread[0] to fix this. >> >> 0: https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagi… <https://opendev.org/openstack/oslo.messaging/src/branch/master/oslo_messagi…> >> >> On 9/17/21 7:00 AM, Adam Tomas wrote: >>> Hi, >>> after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: >>> https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794> <https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794>> >>> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.” >>> but… It is reported :( And produces tons of logs. >>> I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( >>> [oslo_messaging_rabbit] >>> heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 >>> How to avoid this disconnections?? Or what to do to avoid writing this „errors”to logs? >>> Best regards >>> Adam Tomas >>> /var/log/kolla/rabbitmq/rabbitXXX.log >>> 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 >>> 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 >>> 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 >>> 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 >>> 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b >>> 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 >>> 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): >>> missed heartbeats from client, timeout: 60s >>> 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed >>> 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 >>> 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) >>> 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e >>> 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' >>> 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): >>> missed heartbeats from client, timeout: 60s >>> /var/log/kolla/glance/glance-api.log >>> 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 >>> 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 >>> 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 >>> 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 >>> 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 >>> 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 >>> 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 >>> 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 >>> /var/log/kolla/keystone/keystone.log >>> 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> /var/log/kolla/nova/nova-compute.log >>> 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. >>> 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. >>> 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. >>> 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. >>> 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. >>> 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. >>> 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer >>> 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection >>> 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446. > > > > -- > Hervé Beraud > Senior Software Engineer at Red Hat > irc: hberaud > https://github.com/4383/ <https://github.com/4383/> > https://twitter.com/4383hberaud <https://twitter.com/4383hberaud> >

4 4

[all][tc] Technical Committee next weekly meeting on Sept 23rd at 1500 UTC
by Ghanshyam Mann 22 Sep '21

22 Sep '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for Sept 23rd at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, Sept 22th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[stable] remove formal 'unmaintained' phase from EM to EOL transition
by Előd Illés 22 Sep '21

22 Sep '21

Hi stable maintainers! On the Xena Virtual PTG there was a discussion about 'Stable Extended Maintenance to End of Life transition' [1] and at some point we touched that the listed 'Unmaintained' phase [2] does not add value in the process (rather some confusion) so that was agreed that it should be removed (line 397). I've seen that the removal was not yet proposed so I created a patch [3] for it now. So this is just a warning for the change. Thanks, Előd [1] Line 342-397 https://etherpad.opendev.org/p/tc-xena-ptg [2] https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… [3] https://review.opendev.org/c/openstack/project-team-guide/+/810499

1 0

[heat] Proposing Brendan Shephard for heat core
by Rico Lin 22 Sep '21

22 Sep '21

Hi all I would like to suggest to add Brendan Shephard(bshephar(a)redhat.com) to the heat core reviewer group. He has been working on heat actively. provide good quality reviews and commits. I think it will be great to have him as the heat core reviewer. *Rico Lin* OIF Individual Board of directors, OpenStack TC, Multi-arch SIG chair, Heat PTL, Senior Software Engineer@EasyStack

3 2

[cinder] Bug deputy report for week of 2021-22-09
by Sofia Enriquez 22 Sep '21

22 Sep '21

This is a bug report from 2021-08-09 to 2021-15-09. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting ----------------------------------------------------------------------------------------- High - https://bugs.launchpad.net/cinder/+bug/1944118 " cinder-manage db version fails with: AttributeError: 'Engine' object has no attribute 'get_main_option'". Assigned to Stephen Finucane. Medium - https://bugs.launchpad.net/os-brick/+bug/1944474 "iSCSI connections are not reinitiated after host reboot". Assigned to Sophie Huang. - https://bugs.launchpad.net/cinder/+bug/1926412 "When the backup cannot find an available service, update the volume status to none". Assigned to ZhaoYixin - https://bugs.launchpad.net/cinder/+bug/1920729 "PowerStore driver reports no iSCSI targets found". Assigned to Ivan Pchelintsev. Low - https://bugs.launchpad.net/cinder/+bug/1924798 "NetApp ONTAP cannot initialize the driver with SVM scoped mode". Assigned to Felipe Rodrigues. - https://bugs.launchpad.net/cinder/+bug/1924643 "NetApp ONTAP cannot resize a volume LUN with a sub-clone". Assigned to Felipe Rodrigues. Invalid - https://bugs.launchpad.net/cinder/+bug/1922507 "cinder-volume-driver test fails "ModuleNotFoundError: No module named 'fixtures'" Thanks -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[monasca][cloudkitty] [kolla-ansible][victoria] metrics metadata
by Tomasz Rutkowski 22 Sep '21

22 Sep '21

Hi, perhaps I'm missing something - monasca returns metrics metadata when asked for measurements (/metrics/measurements) however cloudkitty asks only for aggregated data (/metrics/statistics). Is there some way to push cloudkitty to work with monasca also by the field-id mappings (metadata values)? with best regards -- Tomasz Rutkowski

1 0

[Outreachy] Call for mentors, project submission by September 26
by Mahati Chamarthy 22 Sep '21

22 Sep '21

Hello! *TL;DR OpenStack is participating in Outreachy for the upcoming December round!* *Please submit projects as soon as possible, the deadline is September 26th, 4 pm UTC:* *https://www.outreachy.org/communities/cfp/openstack/ <https://www.outreachy.org/communities/cfp/openstack/>* Outreachy's goal is to support people from groups underrepresented in the technology industry. Interns will work remotely with mentors from our community. We are seeking mentors to propose projects that Outreachy interns can work on during their internship. If you or your colleagues are interested in mentoring in the next round, please submit your project proposal by *September 26th,* *2021 *over here https://www.outreachy.org/communities/cfp/openstack/ Mentors should read the mentor FAQ https://www.outreachy.org/mentor/mentor-faq and find more details about the Outreachy program and timeline in https://www.outreachy.org/communities/cfp/ . If you want help crafting your project proposal, please contact me Mahati Chamarthy <mahati.chamarthy(a)gmail.com> or Samuel Queiroz < samueldmq(a)gmail.com> and we will be glad to assist you. Thank you, Samuel Queiroz

1 1

[requirements][release][ironic] Exception for ironic deliverables - Xena
by Iury Gregory 21 Sep '21

21 Sep '21

Hello everyone, The ironic CI last week was in a very bad shape and we couldn't merge some patches that we wanted to have in the Xena release, we knew last week was the RC1 and we should have proposed the release for our derivables. Today we were able to merge a workaround to get our CI green again, we will probably have all the required patches merged by EOD or tomorrow morning. I would like to ask for an exception so we can release our deliverables tomorrow. Thanks! -- *Att[]'sIury Gregory Melo Ferreira * *MSc in Computer Science at UFCG* *Part of the ironic-core and puppet-manager-core team in OpenStack* *Software Engineer at Red Hat Czech* *Social*: https://www.linkedin.com/in/iurygregory *E-mail: iurygregory(a)gmail.com <iurygregory(a)gmail.com>*

3 2

Constant HA Virtual Router Flapping
by Justin Moravec 21 Sep '21

21 Sep '21

Any ideas welcome, Kolla-ansible Ussuri deployment. Have some projects that are using virtual routers that are in HA setup that are constantly flapping (flipping between one neutron node to another) causing disconnects. This only happens in SOME projects, other projects with HA routers are working as intended. This only affects HA router deployments, non HA routers work as intended no matter the neutron node they are on. Have verified that each virtual router can reach the external network gateway while it is on each of the neutron nodes. Thanks, Justin

1 0

[neutron] Bug deputy report (Sep 13 - 19)
by Akihiro Motoki 21 Sep '21

21 Sep '21

Hi, Here is my bug deputy report last week. Sorry for late. [Summary] * There are several RC2 candidate bugs. I added a DVR one to Xena RC2 milestone. * dynamic-routing and vpnaas bugs need to be investigated. I would be nice if someone can look into the dynamic-routing one. Regarding vpnaas ones, unfortunately I had no time to check them last week but I will visit them this week. [RC candidates] * https://bugs.launchpad.net/neutron/+bug/1943714 DB session commit error in resource_registry.set_resources_dirty In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943930 [DHCP] new line symbol in opt_name of extra_dhcp_opt causes dnsmasq to fail In Progress, High, Fix already approved * https://bugs.launchpad.net/neutron/+bug/1943846 Create floating agent gw port for DVR fails In Progress, High, Fix already approved NOTE: It is caused by the change in the notification payload during Xena, so I targeted to Xena-RC2. [QoS] * https://bugs.launchpad.net/neutron/+bug/1943724 Change a QoS policy on a bound port does not handle minimum bandwidth direction change In Progress, Medium NOTE: A fix is proposed but it is in the feature patch series of the min pps rule. Hopefully someone familiar with QoS can check the dependency of the fix. [dynamic-routing] * https://bugs.launchpad.net/neutron/+bug/1943725 Automatic cleanup of BGP speakers is too aggressive New, Unassigned [VPNaaS] * https://bugs.launchpad.net/neutron/+bug/1943449 VPNaaS reconfiguration creates duplicate IPtables rules causes the VPN connection to remain DOWN New, Unassigned * https://bugs.launchpad.net/neutron/+bug/1943716 vpn services / vpn connections are stuck in PENDING CREATE New, Unassigned NOTE: RabbitMQ topic mismatch is reported here. It is interesting. I will check later this week. Thanks, Akihiro Motoki (irc: amotoki)

1 0

[release] Release countdown for week R-2, Sep 20 - Sep 24
by Herve Beraud 21 Sep '21

21 Sep '21

Development Focus ----------------- At this point we should have release candidates (RC1 or recent intermediary release) for all the Xena deliverables. Teams should be working on any release-critical bugs that would require another RC or intermediary release before the final release. Actions ------- Early in the week, the release team will be proposing stable/xena branch creation for all deliverables that have not branched yet, using the latest available Xena release as the branch point. If your team is ready to go for creating that branch, please let us know by leaving a +1 on these patches. If you would like to wait for another release before branching, you can -1 the patch and update it later in the week with the new release you would like to use. By the end of the week the release team will merge those patches though, unless an exception is granted. Once stable/xena branches are created, if a release-critical bug is detected, you will need to fix the issue in the master branch first, then backport the fix to the stable/xena branch before releasing out of the stable/xena branch. After all of the cycle-with-rc projects have branched we will branch devstack, grenade, and the requirements repos. This will effectively open them up for Yoga development, though the focus should still be on finishing up Xena until the final release. For projects with translations, watch for any translation patches coming through and merge them quickly. A new release should be produced so that translations are included in the final Xena release. Finally, now is a good time to finalize release notes. In particular, consider adding any relevant "prelude" content. Release notes are targeted for the downstream consumers of your project, so it would be great to include any useful information for those that are going to pick up and use or deploy the Xena version of your project. Upcoming Deadlines & Dates -------------------------- Final RC deadline: September 30 (R-1 week) Final Xena release: October 6 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

Changing Default Network Quotas in Openstack
by Karera Tony 20 Sep '21

20 Sep '21

Hello Team, I have a working Wallaby Openstack deployed using kolla-ansible. However, I have not been able to find a way of changing the Network quotas as I need to increase the defaults. Can someone kindly advise on how to go about it ? Regards Tony Karera

3 2

OpenInfra Live - September 23 at 9am CT
by Ashlee Ferguson 20 Sep '21

20 Sep '21

Hi everyone, This week’s OpenInfra Live episode is brought to you by the OpenInfra Community. Join the volunteer organizers of OpenInfra Days to hear about the happenings at their latest local events, what they're looking forward to in the future, and learn what's happening with OpenInfra all over the world. Episode: OpenInfra Days 2021 Recap Date and time: September 23, 2021 at 9am CT (1400 UTC) You can watch us live on: YouTube: https://www.youtube.com/watch?v=-s1eJa6u_2A <https://www.youtube.com/watch?v=-s1eJa6u_2A> LinkedIn: https://www.linkedin.com/feed/update/urn:li:ugcPost:6844374141978734592/ <https://www.linkedin.com/feed/update/urn:li:ugcPost:6844374141978734592/> Facebook: https://www.facebook.com/104139126308032/posts/4341771172544785/ <https://www.facebook.com/104139126308032/posts/4341771172544785/> WeChat: recording will be posted on OpenStack WeChat after the live stream Speakers: Akihiro Hasegawa, Cloud Operator Days Tokyo Organizer Rico Lin, OpenInfra Days Asia Organizer Sartika Lestari, OpenInfra Days Indonesia Organizer Have an idea for a future episode? Share it now at ideas.openinfra.live <https://openinfrafoundation.formstack.com/forms/openinfralive>. Register now for OpenInfra Live: Keynotes, a special edition of OpenInfra Live on November 17-18th starting at 1500 UTC: https://openinfralivekeynotes.eventbrite.com/ <https://openinfralivekeynotes.eventbrite.com/> Thanks, Ashlee

1 0

[openstack-cli] Missing volume backend commands
by Laurent Dumont 20 Sep '21

20 Sep '21

Hey everyone, Trying to use the volume backend command to get some information regarding storage pools. https://docs.openstack.org/python-openstackclient/latest/cli/command-object… Not sure if it's something on my side, but I cannot see to access a subsection of the volume commands from the CLI client. openstack volume backend pool list openstack: 'volume backend pool list' is not an openstack command. See 'openstack --help'. Did you mean one of these? volume attachment complete volume attachment create volume attachment delete volume attachment list volume attachment set volume attachment show volume backup create volume backup delete volume backup list /usr/local/lib/python3.9/site-packages/openstack/ drwxr-xr-x 41 laurentdumont admin 1.3K 2 Jun 17:03 . drwxr-xr-x 187 laurentdumont admin 5.8K 20 Sep 09:47 .. -rw-r--r-- 1 laurentdumont admin 2.4K 2 Jun 17:03 __init__.py -rw-r--r-- 1 laurentdumont admin 1.2K 2 Jun 17:03 __main__.py drwxr-xr-x 15 laurentdumont admin 480B 2 Jun 17:03 __pycache__ -rw-r--r-- 1 laurentdumont admin 1.4K 2 Jun 17:03 _hacking.py -rw-r--r-- 1 laurentdumont admin 4.6K 2 Jun 17:03 _log.py -rw-r--r-- 1 laurentdumont admin 5.6K 2 Jun 17:03 _services_mixin.py drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 accelerator drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 baremetal drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 baremetal_introspection drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 block_storage drwxr-xr-x 28 laurentdumont admin 896B 2 Jun 17:03 cloud drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 clustering drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 compute drwxr-xr-x 14 laurentdumont admin 448B 2 Jun 17:03 config -rw-r--r-- 1 laurentdumont admin 21K 2 Jun 17:03 connection.py drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 database drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 dns -rw-r--r-- 1 laurentdumont admin 8.6K 2 Jun 17:03 exceptions.py drwxr-xr-x 5 laurentdumont admin 160B 2 Jun 17:03 fixture -rw-r--r-- 1 laurentdumont admin 1.6K 2 Jun 17:03 format.py drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 identity drwxr-xr-x 11 laurentdumont admin 352B 2 Jun 17:03 image drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 instance_ha drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 key_manager drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 load_balancer drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 message drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 network drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 object_store drwxr-xr-x 8 laurentdumont admin 256B 2 Jun 17:03 orchestration drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 placement -rw-r--r-- 1 laurentdumont admin 29K 2 Jun 17:03 proxy.py -rw-r--r-- 1 laurentdumont admin 0B 2 Jun 17:03 py.typed -rw-r--r-- 1 laurentdumont admin 88K 2 Jun 17:03 resource.py -rw-r--r-- 1 laurentdumont admin 13K 2 Jun 17:03 service_description.py drwxr-xr-x 6 laurentdumont admin 192B 2 Jun 17:03 shared_file_system drwxr-xr-x 9 laurentdumont admin 288B 2 Jun 17:03 tests -rw-r--r-- 1 laurentdumont admin 12K 2 Jun 17:03 utils.py -rw-r--r-- 1 laurentdumont admin 641B 2 Jun 17:03 version.py drwxr-xr-x 7 laurentdumont admin 224B 2 Jun 17:03 workflow usr/local/…/site-packages/openstack % python3 Python 3.9.2 (default, Mar 26 2021, 23:27:12) [Clang 12.0.0 (clang-1200.0.32.29)] on darwin Type "help", "copyright", "credits" or "license" for more information. >>> I cannot see the volume folder (but it's on the opendev) The command was added here https://opendev.org/openstack/python-openstackclient/commit/9647d43bd5e77c8… Removing + reinstall the client doesn't seem to work either. Any clues?

2 2

Install controller computehci on 3 nodes with tripleO
by wodel youchi 20 Sep '21

20 Sep '21

Hi, Is it technically possible to install openstack tripleO on 3 nodes, such that all 3 nodes play the roles of controller compute and ceph storage (hci) at the same time, with replication and HA and all? Regards.

3 4

[Swift] Rebalancing EC question
by Reid Guyett 20 Sep '21

20 Sep '21

Hello, We were working on expanding one of our clusters (Ussuri on Ubuntu 18.04) and are wondering about the rebalance behavior of swift-ring-builder. When we run it in debug mode on a 15/4 EC ring, we see this message about "Unable to finish rebalance plan after 2 attempts" and are seeing 100% partitions reassigned. DEBUG: Placed 10899/2 onto dev r1z3-10.40.48.72/d10 DEBUG: Placed 2183/3 onto dev r1z5-10.40.48.76/d11 DEBUG: Placed 1607/1 onto dev r1z3-10.40.48.70/d28 DEBUG: Assigned 32768 parts DEBUG: Gather start is 10278 (Last start was 25464) DEBUG: Unable to finish rebalance plan after 2 attempts Reassigned 32768 (100.00%) partitions. Balance is now 63.21. Dispersion is now 0.00 ------------------------------------------------------------------------------- NOTE: Balance of 63.21 indicates you should push this ring, wait at least 1 hours, and rebalance/repush. ------------------------------------------------------------------------------- Moving 100% seems scary, what does that mean in this situation? Is this message because 1 fragment from every partition is moved and that is the most that it can do per rebalance because they are technically the same partition? When we compare the swift-ring-builder output (partitions per device) between rebalances we can see some partitions move each time until we no longer see the push/wait/rebalance message again. So it's not really moving 100% partitions. Reid

2 2

FreeNAS intergration witn Openstack
by open infra 20 Sep '21

20 Sep '21

Hi, Is it possible to use a freenas storage instead of ceph cluster? Regards, Danishka

3 2

[TripleO] Installation of Undercloud machine on Centos 8 fails in prefetching the Images
by Anirudh Gupta 20 Sep '21

20 Sep '21

Hi Team, I was trying to install Undercloud on Centos 8 using the link below, but facing error in executing "openstack undercloud install" command. The error is in prefetch image Steps followed: 1. Created a non root stack user - sudo useradd stack - sudo passwd stack # specify a password - echo "stack ALL=(root) NOPASSWD:ALL" | sudo tee -a /etc/sudoers.d/stack - sudo chmod 0440 /etc/sudoers.d/stack - su - stack 2. Install the python-tripleo-repos - sudo dnf install -y https://trunk.rdoproject.org/centos8/component/tripleo/current/python3-trip… - In order to install victoria release, executed the command - *sudo -E tripleo-repos -b victoria current ceph* 3. As per the document, "*The remaining repositories configuration steps below should not be done for stable releases!"* - *So, I didn't executed the command* - "sudo -E tripleo-repos current-tripleo-dev ceph" - *Is my understanding correct? This step has to be skipped in case we are going with a stable victoria release?* 4. Modified undercloud.conf file and executed the command - openstack undercloud install *ISSUE:* 2021-09-20 11:16:18,724 p=59556 u=root n=ansible | 2021-09-20 11:16:18.723422 | 525400a8-fc39-4771-208c-000000000e71 | FATAL | Pre-fetch all the containers | undercloud | item= quay.io/tripleovictoria/openstack-swift-account:current-tripleo | error={"ansible_loop_var": "prefetch_image", "changed": false, "msg":" *Failed to pull image quay.io/tripleovictoria/openstack-swift-account:current-tripleo <http://quay.io/tripleovictoria/openstack-swift-account:current-tripleo>", "prefetch_image": "quay.io/tripleovictoria/openstack-swift-account:current-tripleo <http://quay.io/tripleovictoria/openstack-swift-account:current-tripleo>"}* 5. In order to eliminate the problem, I manually installed the image - - sudo podman pull *quay.io/tripleovictoria/openstack-swift-account:current-tripleo <http://quay.io/tripleovictoria/openstack-swift-account:current-tripleo>* After re-executing, then it again failed for the keystone image, which was already installed on the system. I verified in *sudo podman images* command *I tried running this 5-6 times and issue generates randomly for any openstack image ( sometimes for rabbitmq sometimes for swift, keystone) which is already installed in machine* Can someone please help or suggest what step I am missing Note: This I have tried using Centos 8 Stream and Centos 8.4(using --no-stream), but same issue is observed in both Regards Anirudh Gupta

1 0

[keystone] Ed25519 support
by Eduardo Santos 20 Sep '21

20 Sep '21

Hi folks, I'm trying to create an instance with an assigned keypair that uses the Ed25519 algorithm, but I'm unable to login into the instance without providing a password (I generated the public key without a passphrase). RSA keypairs work fine. Is Ed25519 not supported? I didn't find a support matrix in the documentation. Thanks.

2 1

RabbitMQ annoying disconnections
by Adam Tomas 20 Sep '21

20 Sep '21

Hi, after some struggling I have almost „clear” logs (clear=error free :) ) Almost…. RabbitMQ keeps disconnecting sessions and there is a huge amount of disconnect errors in all logs (see below). I found this bug description: https://bugzilla.redhat.com/show_bug.cgi?id=1711794 <https://bugzilla.redhat.com/show_bug.cgi?id=1711794> in which we can read as follows: "this is a recoverable issue that is already handled by how oslo.messaging is designed. disconnection is not an error and should not be reported as such in the logs.” but… It is reported :( And produces tons of logs. I tried to modify heartbeat values - helped a little bit, but I had to increase [database] max_pool_size = 5 and that of course multiplied number of disconnection errors by 5 :( [oslo_messaging_rabbit] heartbeat_timeout_threshold = 720 heartbeat_interval = 360 heartbeat_rate = 4 How to avoid this disconnections?? Or what to do to avoid writing this „errors”to logs? Best regards Adam Tomas /var/log/kolla/rabbitmq/rabbitXXX.log 2021-09-17 06:43:08.771 [error] <0.16056.678> closing AMQP connection <0.16056.678> (x.x.x.x00:44882 -> x.x.x.x00:5672 - mod_wsgi:959:aaa12af5-bb0b-456e-a129-a466eccc24f0): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:08.773 [info] <0.7412.679> Closing all channels from connection 'x.x.x.x00:44882 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.282 [info] <0.4574.679> accepting AMQP connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) 2021-09-17 06:43:09.283 [info] <0.28410.678> accepting AMQP connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) 2021-09-17 06:43:09.285 [info] <0.28410.678> Connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394 2021-09-17 06:43:09.285 [info] <0.4574.679> Connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93 2021-09-17 06:43:09.286 [info] <0.4574.679> connection <0.4574.679> (x.x.x.x11:58170 -> x.x.x.x00:5672 - magnum-conductor:967:08f29ce5-d6ee-43ae-87a9-e8c8856f8c93): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.286 [info] <0.28410.678> connection <0.28410.678> (x.x.x.x11:58172 -> x.x.x.x00:5672 - magnum-conductor:954:c05a1221-6915-4414-ba93-7db5450ff394): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.504 [error] <0.14465.678> closing AMQP connection <0.14465.678> (x.x.x.x00:44914 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.507 [info] <0.14985.679> Closing all channels from connection 'x.x.x.x00:44914 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:09.521 [info] <0.11574.678> accepting AMQP connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) 2021-09-17 06:43:09.523 [info] <0.11574.678> Connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672) has a client-provided name: mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3 2021-09-17 06:43:09.525 [info] <0.11574.678> connection <0.11574.678> (x.x.x.x00:46834 -> x.x.x.x00:5672 - mod_wsgi:964:ced74779-2794-40f6-a01e-276bceb5d7b3): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.654 [info] <0.31575.679> accepting AMQP connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) 2021-09-17 06:43:09.656 [info] <0.31575.679> Connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7 2021-09-17 06:43:09.656 [info] <0.31180.678> accepting AMQP connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) 2021-09-17 06:43:09.658 [info] <0.31575.679> connection <0.31575.679> (x.x.x.x01:55304 -> x.x.x.x00:5672 - magnum-conductor:7:5a1cc3cc-150d-417a-9652-1279597c3db7): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.658 [info] <0.31180.678> Connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b 2021-09-17 06:43:09.660 [info] <0.2757.679> accepting AMQP connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) 2021-09-17 06:43:09.660 [info] <0.31180.678> connection <0.31180.678> (x.x.x.x01:55306 -> x.x.x.x00:5672 - magnum-conductor:954:c8deffa4-7d67-46be-9b9d-09fa20544a2b): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.662 [info] <0.2757.679> Connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672) has a client-provided name: magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315 2021-09-17 06:43:09.663 [info] <0.2757.679> connection <0.2757.679> (x.x.x.x11:58174 -> x.x.x.x00:5672 - magnum-conductor:7:cebf81c9-95f6-48fa-8bde-30c01243d315): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:09.814 [error] <0.22797.678> closing AMQP connection <0.22797.678> (x.x.x.x05:57676 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:09.817 [info] <0.7903.679> Closing all channels from connection 'x.x.x.x05:57676 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.319 [error] <0.18268.678> closing AMQP connection <0.18268.678> (x.x.x.x00:44924 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): missed heartbeats from client, timeout: 60s 2021-09-17 06:43:10.322 [info] <0.28032.678> Closing all channels from connection 'x.x.x.x00:44924 -> x.x.x.x00:5672' because it has been closed 2021-09-17 06:43:10.330 [info] <0.19596.678> accepting AMQP connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) 2021-09-17 06:43:10.333 [info] <0.19596.678> Connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672) has a client-provided name: nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09 2021-09-17 06:43:10.334 [info] <0.19596.678> connection <0.19596.678> (x.x.x.x00:46850 -> x.x.x.x00:5672 - nova-conductor:957:e3951cbd-37ca-4d9f-8fac-394c83b36f09): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:10.831 [info] <0.9838.678> accepting AMQP connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) 2021-09-17 06:43:10.834 [info] <0.9838.678> Connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672) has a client-provided name: neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e 2021-09-17 06:43:10.835 [info] <0.9838.678> connection <0.9838.678> (x.x.x.x05:57878 -> x.x.x.x00:5672 - neutron-openvswitch-agent:7:78d30277-173a-47d4-b8a8-3edde75f115e): user 'openstack' authenticated and granted access to vhost '/' 2021-09-17 06:43:11.738 [error] <0.15275.678> closing AMQP connection <0.15275.678> (x.x.x.x01:53414 -> x.x.x.x00:5672 - mod_wsgi:968:b60f723c-882e-4780-a162-55c03ebe6179): missed heartbeats from client, timeout: 60s /var/log/kolla/glance/glance-api.log 2021-09-17 13:17:23.212 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:29.208 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:17:42.190 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:17:42] "GET / HTTP/1.1" 300 1446 0.001949 2021-09-17 13:17:46.351 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:12.336 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:12] "GET / HTTP/1.1" 300 1446 0.003754 2021-09-17 13:18:13.703 956 INFO eventlet.wsgi.server [req-1a00db34-349b-4ed3-8625-edfadc939be4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:13] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.139229 2021-09-17 13:18:33.065 955 INFO eventlet.wsgi.server [req-a57046eb-549e-4837-95c7-d0994c05449e d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.096302 2021-09-17 13:18:33.077 955 INFO eventlet.wsgi.server [req-3a8af281-6732-4afd-b3ec-e502367bce51 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:18:33] "GET /v2/schemas/image HTTP/1.1" 200 6259 0.003982 2021-09-17 13:18:36.747 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:36.759 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:42.460 957 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:18:42] "GET / HTTP/1.1" 300 1446 0.001383 2021-09-17 13:18:52.659 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:18:52.670 956 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:12.598 955 INFO eventlet.wsgi.server [-] x.x.x.x00 - - [17/Sep/2021 13:19:12] "GET / HTTP/1.1" 300 1446 0.001590 2021-09-17 13:19:16.367 957 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.127 955 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:27.184 955 INFO eventlet.wsgi.server [req-8f68efc9-f46a-450d-8138-928e4f2585b4 d33b56f82a984f6a99bad58717f6f0ef 6ee7e3517d254f9e9ee5eac7d56a3179 - default default] x.x.x.x09,x.x.x.x04 - - [17/Sep/2021 13:19:27] "GET /v2/images?limit=20 HTTP/1.1" 200 2990 0.032917 /var/log/kolla/keystone/keystone.log 2021-09-17 13:13:54.670 958 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.148 960 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.150 962 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:55.345 959 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:56.044 961 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:18.402 966 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:31.372 963 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:36.580 967 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:38.771 965 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:12.226 964 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer /var/log/kolla/nova/nova-compute.log 2021-09-17 13:10:07.710 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56226. 2021-09-17 13:11:11.473 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.046 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:11:34.721 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:13:07.699 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:13:08.734 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56434. 2021-09-17 13:14:11.501 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.070 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:14:34.745 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:16:08.724 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:16:09.756 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56638. 2021-09-17 13:17:34.098 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:19:09.747 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:19:10.781 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 56834. 2021-09-17 13:20:34.124 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:34.777 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:20:51.789 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:22:10.772 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:22:11.806 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57038. 2021-09-17 13:23:34.149 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:34.803 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:23:51.818 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:25:11.795 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:25:12.842 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57240. 2021-09-17 13:26:34.176 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:26:34.830 7 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer 2021-09-17 13:28:12.812 7 ERROR oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] AMQP server on x.x.x.x00:5672 is unreachable: Server unexpectedly closed connection. Trying again in 1 seconds.: OSError: Server unexpectedly closed connection 2021-09-17 13:28:13.848 7 INFO oslo.messaging._drivers.impl_rabbit [-] [9d51e0ec-bd43-46b4-8c57-a9c6aa01903d] Reconnected to AMQP server on x.x.x.x00:5672 via [amqp] client with port 57446.

4 5

[ops][cinder] is anyone using RateLimitingMiddleware ?
by Brian Rosmaita 20 Sep '21

20 Sep '21

Hello operators, Takashi Kajinami noticed that cinder's RateLimitingMiddleware class lives among the legacy v2 classes and should be moved to live with the v3 classes, but then it occurred to him that maybe it should simply be deprecated and removed [0]. There are a few reasons to believe that nobody is actually using this functionality, for example, it's not included in the default api-paste.ini file, this middleware no longer exists in Nova, it gives you inexact behavior with multiple API nodes, etc. We discussed this topic at today's cinder meeting and decided to ask whether anyone is using the RateLimitingMiddleware with cinder before deprecating it for removal. If it's going to be deprecated, we'd like to do it in Xena, so please reply to this email or leave a comment on the deprecation patch [1] (or preferably both) before 23:59 UTC on Tuesday 14 September 2021. Thank you! [0] https://bugs.launchpad.net/cinder/+bug/1942696 [1] https://review.opendev.org/c/openstack/cinder/+/807469

2 1

Edge for scientific research session on September 13 Edge WG meeting
by Ildiko Vancsa 19 Sep '21

19 Sep '21

Hi, I’m very excited to invite you to a session with the Chameleon project to discuss how edge computing is powering education, scientific research and more! As always in the area of edge there are challenges and gaps that we need to solve together. In that sense the session will start with a presentation that we will then turn into a discussion to turn towards discussing solutions. Please also note that the Chameleon project is using OpenStack to build edge solutions. As they identified some gaps they would like to work with the community to address the challenges that they’ve been hitting. Please join us to get their feedback and find the best way to improve OpenStack to deliver a platform that can be used on the edge! The session will take place on the OpenInfra Edge Computing Group weekly call on __September 13, at 1300 UTC / 6am Pacific__. For the latest dial-in information please refer to this page: https://wiki.openstack.org/wiki/Edge_Computing_Group#Meetings Please find the details of the session below. Thanks and Best Regards, Ildikó Title: Chameleon: a cloud to edge infrastructure for science Abstract: The NSF-funded Chameleon project (www.chameleoncloud.org) provides an OpenStack-based bare metal reconfigurable cloud that supports systems research, education, and emergent application projects in science. The facility has been in existence for 6+ years, and has served ~6,000 users working on a variety of projects ranging from networking, operating systems design, and security, to innovative applications in astronomy, physics, and life sciences. Recent research trends in IoT and edge computing, as evidenced by increased numbers of project applications in those areas, made it clear that Chameleon needed to expand into the edge space. Remaining within the OpenStack framework allowed us to leverage end-user features that we developed for Chameleon, such as access via federated identity and Jupyter integration. Accordingly, we developed infrastructure called CHI@Edge that adapts Zun to provision containers on edge devices and significantly expands its capabilities to work in the edge space. The talk will describe our use case and the emergent product definition, the OpenStack adaptations we made to the current preview implementation, the directions we plan to take in our further work, as well as the ecosystem of projects and NSF infrastructure deployments leveraging this work and collaborating with us.

3 9

OVS_AFXDP for production
by Satish Patel 18 Sep '21

18 Sep '21

Folks, I’m running some high performance workload on SRIOV and planning to move from SRIOV to DPDK but recently Hearing a lots about AFDXP data path and just wanted to check with expert, is this ready for production? How good it is to compare with DPDK? Sent from my iPhone

1 0

[Interop] New meeting time
by Arkady Kanevsky 17 Sep '21

17 Sep '21

The interop WG changed its weekly meeting time to Thursday 16:00pm UTC. The Google meet and etherpad stay the same. See https://wiki.openstack.org/wiki/Governance/InteropWG#Meetings Thanks, Arkady -- Arkady Kanevsky, Ph.D. Phone: 972 707-6456 Corporate Phone: 919 729-5744 ext. 8176456

1 0

[all][tc][goals] Project Specific PTL and Contributor Documentation: Week R-20 Update
by Ghanshyam Mann 17 Sep '21

17 Sep '21

Hello Everyone, As you know, we did not select any community-wide goal for Xena cycle and in PTG, TC decided to spend more time on the previous cycle pending community-wide goal work[1] One of the previous cycle (Ussuri) goals is 'Project Specific PTL and Contributor Documentation' which still not completed by many projects. - https://governance.openstack.org/tc/goals/selected/ussuri/project-ptl-and-c… I am starting the work to finish this goal which is pretty straight forwards and sending the updates in this ML thread. Please review the patches and merge them or push the patches for your project's repo if not yet done. Gerrit Topic: https://review.opendev.org/q/topic:%22project-ptl-and-contrib-docs%22+(stat… Tracking: https://storyboard.openstack.org/#!/story/2007236 Progress Summary: =============== * Projects completed: 32 * Projects for which patches are up and required to merge those: 10 * Projects for which patches are not yet up: 12 [1] https://etherpad.opendev.org/p/tc-xena-tracker (Item#2) -gmann

2 2

[all][tc] What's happening in Technical Committee: summary 17th Sept, 21: Reading: 5 min
by Ghanshyam Mann 17 Sep '21

17 Sep '21

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * TC this week IRC meeting held on Sept 16th Thursday. * Most of the meeting discussions are summarized in the below sections (Completed or in-progress activities section). To know more details, you can check the complete logs @ https://meetings.opendev.org/meetings/tc/2021/tc.2021-09-16-15.00.log.html * We will have next week's meeting on Sept 23rd, Thursday 15:00 UTC on IRC, feel free the topic in agenda [1] by Sept 22nd. 2. What we completed this week: ========================= * PTLs for all the six leaderless projects are now assigned and merged[2] * Added openstack-loadbalancer charm and interfaces[3] * TC chair and vice-chair for Yoga cycle are selected. I will continue as chair[3] and yoctozepto as vice-chair[4]. 3. Activities In progress: ================== TC Tracker for Xena cycle ------------------------------ TC is using the etherpad[6] for Xena cycle working item. We will be checking and updating the status biweekly in the same etherpad. Open Reviews ----------------- * Three open reviews for ongoing activities[7]. TC tags analysis ------------------- * As discussed in the last PTG, TC is working on an analysis of the usefulness of the Tags framework[8] or what all tags can be cleaned up. * It seems no response yet on email from yoctozepto[9], if you are an operator, please respond to the email and based on the feedback we will continue the discussion in PTG. Collecting Pain points from projects/SIGs/pop-up ----------------------------------------------------------- * No updates on this, we will continue discussing it in PTG[10], join us if you are interested to help/discuss this topic. Project updates ------------------- * Add the cinder-netapp charm to Openstack charms[11] * Retiring js-openstack-lib [12] * Retire puppet-freezer[13] Yoga release community-wide goal ----------------------------------------- * Please add the possible candidates in this etherpad [14]. * Current status: "Secure RBAC" is selected for Yoga cycle[15]. PTG planning ---------------- * We are collecting the PTG topics in etherpad[16], feel free to add any topic you would like to discuss. * We discussed the live stream of one of the TC PTG sessions like we did last time. Once we will have more topics in etherpad then we can select the appropriate one. Test support for TLS default: ---------------------------------- * Rico has started a separate email thread over testing with tls-proxy enabled[17], we encourage projects to participate in that testing and help to enable the tls-proxy in gate testing. 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[18]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [19] 3. Office hours: The Technical Committee offers a weekly office hour every Tuesday at 0100 UTC [20] 4. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] https://etherpad.opendev.org/p/yoga-leaderless [3] https://review.opendev.org/c/openstack/governance/+/807837 [4] https://review.opendev.org/c/openstack/governance/+/807163 [5] https://review.opendev.org/c/openstack/governance/+/807178 [6] https://etherpad.opendev.org/p/tc-xena-tracker [7] https://review.opendev.org/q/project:openstack/governance+status:open [8] https://governance.openstack.org/tc/reference/tags/index.html [9] http://lists.openstack.org/pipermail/openstack-discuss/2021-September/02480… [10] https://etherpad.opendev.org/p/pain-point-elimination [11] https://review.opendev.org/c/openstack/governance/+/809011 [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] https://review.opendev.org/c/openstack/governance/+/807163 [14] https://etherpad.opendev.org/p/y-series-goals [15] https://review.opendev.org/c/openstack/governance/+/803783 [16] https://etherpad.opendev.org/p/tc-yoga-ptg [17] http://lists.openstack.org/pipermail/openstack-discuss/2021-June/023000.html [18] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [19] http://eavesdrop.openstack.org/#Technical_Committee_Meeting [20] http://eavesdrop.openstack.org/#Technical_Committee_Office_hours -gmann

1 0

[release][PTL] broken gates and stuck releases
by Herve Beraud 17 Sep '21

17 Sep '21

Hello PTLs, For your information our gates are currently broken so we are not able to merge patches and so to release deliverables. indeed openstack/release rely on openstack/governance to validate releases. The governance repo relies on pybot2 which is no longer maintained and became not installable a few hours ago. That broke the release gate and led us to a catch-22 situation where we are not able to release the governance repo to fix our repo. Our current solution is to pull openstack/governance from git source instead of from pypi. By doing that we will pull the fixed version from opendev git and we will unblock our gates. In this way, we will be able to restart to merge patches and release deliverables. We currently wait for some feedback from the infra team to help us to force merging the needed fixes by using their gerrit super admin rights, hence, bypassing the catch-22 situation. This situation highlighted the cross dependency between these two repos (governance and releases) and I don't think that this is something healthy. I propose to keep pulling the governance repo from git source even after we will fix the catch-22 situation. Any opinions? Thanks for your understanding, we will inform you when we will have updates. https://opendev.org/openstack/governance/commit/9128e502253db9aac346ce25726… https://review.opendev.org/c/openstack/releases/+/809626 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

3 3

Delete Volume on Instance Delete
by Derek O keeffe 17 Sep '21

17 Sep '21

Hi all, I was able to set "Create new volume" to no by default on the launch instance page by adding the following to /etc/openstack-dashboard/local_settings.py LAUNCH_INSTANCE_DEFAULTS ={ 'create_volume': False,} Is there anyway I can set "Delete Volume on Instance Delete" to yes by default should someone select to create a new volume when launching an instance. I tried the following but no luck LAUNCH_INSTANCE_DEFAULTS ={ 'create_volume': False, 'delete_on_termination': True} Thanks for any help/info in advance. Regards,Derek

2 2

How to resume/abort a migration in error state
by Yingji Sun 17 Sep '21

17 Sep '21

Buddies, I am in openstack train and I have some failure live-migration cases that leave some migration states to error. These failure cases may block the following migration request in “queued” state. Is it possible that I force to complete the failure cases ? live_migrate_force_complete only works for a “running”migration and live_migrate_abort works only for “running/queued/preparing”migration. Any suggestion would be appreciated. Yingji.

1 0

[neutron] Drivers meeting - Friday 17.09.2021 - cancelled
by Lajos Katona 17 Sep '21

17 Sep '21

Hi, Due to lack of topics for Today's Drivers meeting, let's cancel the meeting. Lajos (lajoskatona)

1 0