September 2021 - openstack-discuss

Problems with Fluentd buffer [fluentd][kolla-ansible]
by Adam Tomas 27 Sep '21

27 Sep '21

Hi, after some time of using newly deployed kolla-ansible I have fluentd errors: on all controllers, storage and network nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 01:40:46 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 01:40:47.847564807 +0200 chunk="5ccee7b1762157f0ae10dcbeb4e14fd8" error_class=RestClient::GatewayTimeout error="504 Gateway Timeout" /var/log/kolla/fluentd/fluentd.log:2021-09-27 04:49:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 04:49:32.728326628 +0200 chunk="5ccf11fdc0d6876abdef813211371285" error_class=RestClient::RequestTimeout error="408 Request Timeout” on compute nodes: /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-statsd.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"openstack_python", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"openstack_python\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"error"=>"#<Fluent::Plugin::Buffer::BufferOverflowError: buffer space has too many data>", "location"=>"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'", "tag"=>"kolla.var.log.kolla.monasca.agent-statsd.log", "message"=>"emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error=\"buffer space has too many data\" location=\"/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'\" tag=\"kolla.var.log.kolla.monasca.agent-statsd.log\""} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="openstack_python" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="fluent.warn" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [warn]: #0 emit transaction failed: error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data" location="/opt/td-agent/lib/ruby/gems/2.7.0/gems/fluentd-1.11.2/lib/fluent/plugin/buffer.rb:293:in `write'" tag="kolla.var.log.kolla.monasca.agent-forwarder.log" /var/log/kolla/fluentd/fluentd.log:2021-09-27 06:41:38 +0200 [error]: #0 failed to emit fluentd's log event tag="fluent.warn" event={"action"=>:throw_exception, "message"=>"failed to write data into buffer by buffer overflow action=:throw_exception"} error_class=Fluent::Plugin::Buffer::BufferOverflowError error="buffer space has too many data” and on monitoring node: 2021-09-27 14:40:50 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=84.05643947119825 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:42:53 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=122.65328024700284 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:44:15 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=82.32426812895574 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:46:05 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:46:05 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:46:06.042150289 +0200 chunk="5ccf89c9ec54b09e616f7167d5f93cc1" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:46:05 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:47:58 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=112.90601075813174 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:47:58 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89c9ec54b09e616f7167d5f93cc1" 2021-09-27 14:50:16 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=136.24769522389397 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:52:27 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=129.86474119895138 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:54:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=100.71324555086903 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:56:31 +0200 [warn]: #0 Bulk sending messages to monasca-api threw exception exceptionew=#<RestClient::Exceptions::ReadTimeout: Timed out reading data from server> 2021-09-27 14:56:31 +0200 [warn]: #0 failed to flush the buffer. retry_time=0 next_retry_seconds=2021-09-27 14:56:32.140741991 +0200 chunk="5ccf89e1feb471b7ce25a26d37977baa" error_class=RestClient::Exceptions::ReadTimeout error="Timed out reading data from server" 2021-09-27 14:56:31 +0200 [warn]: #0 suppressed same stacktrace 2021-09-27 14:58:03 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=90.91785193886608 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 14:58:03 +0200 [warn]: #0 retry succeeded. chunk_id="5ccf89e1feb471b7ce25a26d37977baa" 2021-09-27 15:00:01 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/server.log; waiting 5 seconds 2021-09-27 15:00:01 +0200 [info]: #0 following tail of /var/log/kolla/kafka/server.log 2021-09-27 15:00:10 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=127.54797655600123 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:02:29 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=137.74220423400402 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:08 +0200 [warn]: #0 buffer flush took longer time than slow_flush_log_threshold: elapsed_time=38.81670297612436 slow_flush_log_threshold=20.0 plugin_id="object:bff4" 2021-09-27 15:03:13 +0200 [info]: #0 detected rotation of /var/log/kolla/kafka/controller.log; waiting 5 seconds 2021-09-27 15:03:13 +0200 [info]: #0 following tail of /var/log/kolla/kafka/controller.log What seems to be the problem? Is there any way to run more than one fluentd worker in kolla? Best regards Adam Tomas

1 0

[keystone] Correct policies setup for System Administrators
by Jan Wasilewski 27 Sep '21

27 Sep '21

Hello, I am preparing policies configuration before an upgrade to the newer OpenStack release(Stein) and I would like to create a group of System Administrators to be able to get i.e. a list of all projects in the OpenStack cloud. I was following a description from this page [1] but it seems my admin user is able to get only a list of projects where it is directly added(i.e. with member role, reader role, or admin role). I am just wondering if we can list all of the OpenStack projects by System Administrator user without role reader added to every single project? To summarize what steps were done so far: - Original policy.json file which was used is here [2] - Only one option was changed so far: from: "identity:list_projects": "rule:cloud_admin or rule:admin_and_matching_domain_id", to: "identity:list_projects": "(role:reader and system_scope:all) or (role:reader and domain_id:%(target.domain_id)s)", - Output for command: openstack role assignment list --system all --role member --role reader +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+ | e39e97c23bfe45d1a2f9689b6985f990 | | a0841b83f583477887219f27dd95477b | | | all | False | +----------------------------------+------+----------------------------------+---------+--------+--------+-----------+Shows only role reader, not role member, which is a bit strange if we compare with linked page above. But we have this in implied roles:openstack implied role list +----------------------------------+-----------------+----------------------------------+-------------------+ | Prior Role ID | Prior Role Name | Implied Role ID | Implied Role Name | +----------------------------------+-----------------+----------------------------------+-------------------+ | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 3f20cb7be46346a8b2ba65c4684d50a3 | member | | a3c7bb5d06884b048c1bfb4403b82b42 | admin | 9fe2ff9ee4384b1894a90878d3e92bab | _member_ | | 3f20cb7be46346a8b2ba65c4684d50a3 | member | e39e97c23bfe45d1a2f9689b6985f990 | reader | +----------------------------------+-----------------+----------------------------------+-------------------+- Admin roles are grouped in a group ATM.Admin: openstack role assignment list --names --system all --role admin: +-------+---------------------+-------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +-------+---------------------+-------------------+---------+--------+--------+-----------+ | admin | | ATM.Admin@Default | | | all | False | | admin | admin@Default | | | | all | False | | admin | jwasilewski@Default | | | | all | False | +-------+---------------------+-------------------+---------+--------+--------+-----------+Just to be sure that IDs are linked, we can check it here:openstack role assignment list --system all --role admin +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | Role | User | Group | Project | Domain | System | Inherited | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ | a3c7bb5d06884b048c1bfb4403b82b42 | | a0841b83f583477887219f27dd95477b | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | 19416fe5a2da45c88eb66c3aaf856c73 | | | | all | False | | a3c7bb5d06884b048c1bfb4403b82b42 | f42df418fd404d04b9bdabf2f1b49fd9 | | | | all | False | +----------------------------------+----------------------------------+----------------------------------+---------+--------+--------+-----------+ So by linking roles(implied roles): admin(a3c7bb5d06884b048c1bfb4403b82b42 ) -> member(3f20cb7be46346a8b2ba65c4684d50a3) -> reader(e39e97c23bfe45d1a2f9689b6985f990). Correlation is visible, based on that my user(jwasilewski) should retrieve a full project list, but it seems only three projects are visible where this user is an admin. I do not want to add my user as a reader to every single project to be able to list all of them. Is there a way how to make it or the only way is to add this role(reader) for a user to all projects? Thank you in advance for any suggestions. Best regards, Jan Wasilewski [1] https://docs.openstack.org/keystone/stein/admin/service-api-protection.html… -> https://docs.openstack.org/keystone/latest/admin/service-api-protection.htm… [2] https://paste.openstack.org/show/bq0HgyqouZF1vywKVkGn/

1 0

[neutron] CI update
by Slawek Kaplonski 27 Sep '21

27 Sep '21

Hi, Due to very urgent things which I had yesterday, I wasn't able to run Neutron CI meeting as usually. Fortunatelly we don't have many new issues in our CI. There is only one new issue in our scenario jobs which I wanted to discuss [1]. It's impacting Ironic gates but I noticed it also in the Neutron CI as well. See [2] or [3] for example. I'm not sure about Ironic jobs but in Neutron I saw it mostly (or only, I'm not sure) in the multinode jobs. [1] https://bugs.launchpad.net/neutron/+bug/1944201 [2] https:// e36beaa2ff297ebe7d5f-5944c3d62ed334b8cdf50b534c246731.ssl.cf5.rackcdn.com/ 805849/9/check/neutron-ovs-tempest-dvr-ha-multinode-full/f83fa96/compute1/ logs/screen-q-agt.txt [3] https://storage.bhs.cloud.ovh.net/v1/ AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_88f/803045/12/check/ neutron-ovs-tempest-slow/88f8bb7/job-output.txt -- Slawek Kaplonski Principal Software Engineer Red Hat

5 8

[neutron] Bug deputy report (week starting on 2021-09-20)
by Bernard Cafarelli 27 Sep '21

27 Sep '21

Hello neutrinos, here is the list of bugs reported for last week, overall all bugs have assignees or fixes in progress. Both critical bugs are merged and with pending xena backports (this week is the final RCs week) In details: Critical * neutron-openvswitch-agent crashes on start with firewall config of br-int - https://bugs.launchpad.net/neutron/+bug/1944201 First seen in Ironic jobs but we also see it in our jobs, fix by ralonsoh merged: https://review.opendev.org/c/openstack/neutron/+/810592 (pending xena backport) * neutron_tempest_plugin.scenario.test_trunk.TrunkTest.test_trunk_subport_lifecycle fails with Port already has an attached device - https://bugs.launchpad.net/tripleo/+bug/1943708 Previously reported as https://bugs.launchpad.net/neutron/+bug/1943679, fix with revert proposed by slaweq and merged: https://review.opendev.org/c/openstack/neutron/+/809550 (pending xena backport) High * Traffic leaked from dhcp port before vlan tag is applied - https://bugs.launchpad.net/neutron/+bug/1930414 Possible security issue that was under investigation and now public, assigned to rubasov * OVN does not update the network segment tag - https://bugs.launchpad.net/neutron/+bug/1944708 Assigned to lucasagomes, fix in review: https://review.opendev.org/c/openstack/neutron/+/810704 Medium * Fix OVN metadata agent functional tests for Chassis_Private - https://bugs.launchpad.net/neutron/+bug/1944460 Functional OVN tests failing when using newer OVN, fix merged: https://review.opendev.org/c/openstack/neutron/+/810309 * IPv6 slaac subnet creation causes FixedIpsSubnetsNotOnSameSegment error - https://bugs.launchpad.net/neutron/+bug/1944948 Error raised when creating an IPv6 slaac subnet with multisegment, assigned to elvira Incomplete * Instances with SRIOV ports loose access after failed live migrations - https://bugs.launchpad.net/neutron/+bug/1944619 SRIOV live migration issue, liuyulong gave some pointers -- Bernard Cafarelli

1 0

[release] Release countdown for week R-1, Sep 27 - Oct 1
by Herve Beraud 27 Sep '21

27 Sep '21

Development Focus ----------------- We are on the final mile of the Xena development cycle! Remember that the Xena final release will include the latest release candidate (for cycle-with-rc deliverables) or the latest intermediary release (for cycle-with-intermediary deliverables) available. September 30 is the deadline for final Xena release candidates as well as any last cycle-with-intermediary deliverables. We will then enter a quiet period until we tag the final release on October 6, 2021. Teams should be prioritizing fixing release-critical bugs, before that deadline. Otherwise it's time to start planning the Yoga development cycle, including discussing sessions content in preparation of the PTG on the week of October 18. Actions ------- Watch for any translation patches coming through on the stable/xena branch and merge them quickly. If you discover a release-critical issue, please make sure to fix it on the master branch first, then backport the bugfix to the stable/xena branch before triggering a new release. Please drop by #openstack-release with any questions or concerns about the upcoming release ! Upcoming Deadlines & Dates -------------------------- Final Xena release: October 6 Yoga PTG: October 18-22 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud

1 0

[neutron][ovn] support for stateless NAT for floating ip in ml2 ovn
by Moshe Levi 26 Sep '21

26 Sep '21

Hi all, OVN support stateless NAT operations [1] for use case of 1:1 mapped between inner and external ips, i.e dnat_and_snat rule. In openstack is the floating ip use-case. Looking on ml2 ovn support it seem that it only support floating ip with connection tracking. Can ml2 ovn support also the stateless NAT option? Is there concerns using stateless NAT? [1] - https://patchwork.ozlabs.org/project/openvswitch/cover/1570154179-14525-1-g…

3 8

[requirements] Please consider bumping docker to 5.0.2 in Xena u-c
by Pierre Riteau 26 Sep '21

26 Sep '21

Hello, Xena upper constraints currently contain docker 5.0.1, released on August 31. This version includes a critical issue [1] which was fixed by version 5.0.2 released the next day. Except for a docstring update this is the only change, according to git diff. Would it be possible to update Xena u-c to use docker 5.0.2? We currently have to work around it in Kayobe to avoid installing the buggy version [2]. Thanks, Pierre Riteau (priteau) [1] https://github.com/docker/docker-py/issues/2885 [2] https://review.opendev.org/c/openstack/kayobe/+/807345/9/ansible/kolla-targ…

3 2

[zun][requirements] request to blacklist docker 5.0.1 for X release
by Hongbin Lu 26 Sep '21

26 Sep '21

Hi, It appears that docker 5.0.1 is broken [1]. All the OpenStack projects that depends on docker-py (i.e. Zun [2]) are impacted. Would the requirements team consider blacklist docker 5.0.1 for X release? Please note that Zun must have such blacklist. Otherwise, the X release will be totally broken. [1] https://github.com/docker/docker-py/issues/2885 [2] https://bugs.launchpad.net/zun/+bug/1945088 Best regards, Hongbin

2 2

anyone running ovn with ovs+dpdk
by Satish Patel 25 Sep '21

25 Sep '21

Folks, Anyone running ovn with dpdk on ubuntu 20.04 (Focal) with ovs version 2.13.3 ? I am trying to do that but my ovs switch keep crashing as soon as i spin up vm and it creates socket file in /var/run/openvswitch and I am not able to understand what is wrong with it. (no generating crash dump also) Please advise or share your thoughts.

1 0

All Devstack Based Jobs Failing to Create Swap on master
by Clark Boylan 24 Sep '21

24 Sep '21

Hello everyone, At approximately 12:53 UTC September 24 (today) [0] Devstack's Xena branch was created. Due to an unexpected side effect in Devstack's Zuul configs this has caused all Devstack based jobs against master and possible stable/xena to RETRY_LIMIT and fail. The underlying issue you'll see in the jobs is that mkswap cannot create swap because it already exists [1]. This happens because Devstack's pre-run playbook is running twice [2] and the configure-swap role is not idempotent. The role is running twice because the Devstack base job is loading master and stable/xena's configs and composing them together which results in two pre-run playbooks being selected one for each branch. This happens because Devstack's stable/xena branch explicitly states that its configs apply to master and feature/r1 branches [3]. I've pushed up a change [4] to remove this Zuul configuration from the Xena branch which I expect will fix things for us. [0] https://zuul.opendev.org/t/openstack/build/ebe77aee0c104c9e9b917ff28a4731ee… [1] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [2] https://zuul.opendev.org/t/openstack/build/52fa83b18e1a476b80afc41617fe68f9… [3] https://opendev.org/openstack/devstack/src/branch/stable/xena/.zuul.yaml#L1… [4] https://review.opendev.org/c/openstack/devstack/+/810947

2 2