February 2021 - openstack-discuss

[magnum][heat] Rolling system upgrades
by Krzysztof Klimonda 01 Feb '21

01 Feb '21

Hi, While testing magnum, a problem of upgrades came up - while work has been done to make kubernetes upgrades without interruption, operating system upgrades seem to be handled only partially. According to the documentation, two ways of upgrading system are available: - via specifying ostree_commit or ostree_remote labels in the cluster template used for upgrade - via specifying a new image in the cluster template used for upgrade The first one is specific to Fedora Atomic (and, while probably untested, seems to be mostly working with Fedora CoreOS) but it has some drawbacks. Firstly, due to base image staying the same we require this image for the life of the cluster, even if OS has already been upgraded. Secondly, using this method only upgrades existing instances and new instances (spawned via scaling cluster up) will not be upgraded. Thirdly, even if that is fixed I'm worried that at some point upgrading from old base image to some future ostree snapshot will fail (there is also cost associated with diff growing with each release). The second method, of specifying a new image in the cluster template used for upgrade, comes with an ugly warning about nodes not being drained properly before server rebuild (and it actually doesn't seem to be working anyway as the new image parameter is not being passed to the heat template on upgrade). This does however seem like a more valid approach in general. I'm not that familar with Heat, and the documentation of various OS::Heat::Software* resources seems inconclusive, but is there no way of executing some code before instance is rebuilt? If not, how are other projects and users handling this in general? -- Krzysztof Klimonda kklimonda(a)syntaxhighlighted.com

2 3

[neutron] bug deputy report for week of 2021-01-25
by Rodolfo Alonso Hernandez 01 Feb '21

01 Feb '21

Hello Neutrinos: This is the bug report from 2021-01-25 to 2021-01-31. Critical: - https://bugs.launchpad.net/neutron/+bug/1913718: “Enforcer perform deprecation logic unnecessarily”. Assigned to Slawek. High: - https://bugs.launchpad.net/neutron/+bug/1913401: “Timeout during creation of interface in functional tests”. Unassigned - https://bugs.launchpad.net/neutron/+bug/1913572: “[CI] Error in "tox-docs" job due to an error in library amqp”. Assigned to Rodolfo. - https://bugs.launchpad.net/neutron/+bug/1914037: “scenario tests tempest.scenario.test_network_v6.TestGettingAddress fails”. Unassigned. Medium: - https://bugs.launchpad.net/neutron/+bug/1913297: “Network creation fails when enable_security_group = False with error "Unknown quota resources ['security_group_rule’]”. Assigned to Slawek - https://bugs.launchpad.net/neutron/+bug/1913269: “DHCP entries for floating IP ports are not needed really”. Assigned to Slawek. - https://bugs.launchpad.net/neutron/+bug/1913723: “[DHCP] Race condition during port processing events in DHCP agent”. Assigned to Rodolfo. Low: - https://bugs.launchpad.net/neutron/+bug/1913280: “Keepalived unit tests don't mocks properly _is_keepalived_use_no_track_supported() function”. Assigned to Slawek - https://bugs.launchpad.net/neutron/+bug/1913180: “QoS policy with minimum bandwidth rules can be assigned to a VXLAN network if the network doesn't have any ports”. Unassigned. - https://bugs.launchpad.net/neutron/+bug/1913664: “[CI] neutron multinode jobs does not run neutron_tempest_plugin scenario cases”. Assigned to Mamatisa. Incomplete: - https://bugs.launchpad.net/neutron/+bug/1913038: “Port remains in status DOWN when fields "Device ID" and "Device owner" are filled from dashboard”. Undecided/unconfirmed: - https://bugs.launchpad.net/neutron/+bug/1913621: “Permant ARP entries not added to DVR qrouter when connected to two Networks”. - https://bugs.launchpad.net/neutron/+bug/1913646: “DVR router ARP traffic broken for networks containing multiple subnets”. Probably a duplicate of LP#1913621. Regards.

1 0

Mistral Maintenance
by Renat Akhmerov 01 Feb '21

01 Feb '21

Hi, Just for those who don’t know me, I’m Renat Akhmerov who originally started the project called Mistral (Workflow Service) and I’ve been the PTL of the project most of the time. The project was started in 2013 when I was at Mirantis, then in 2016 I moved to Nokia and continued to work on the project. My activities at Nokia were related with Mistral on 90%. On 25th of January I joined the company called Tailify where I won’t be contributing to Mistral officially anymore. On that note, I want to say that for now I’ll try to provide minimally required maintenance (reviews, releases, CI fixes) and consulting but I don’t think I’ll continue to be the major contributor of the project. And, most likely, I’ll stop working on the project completely at some point. Don’t know when though, it depends on my new load, my capacity and desire. I still care about the project and want it to live further. So, if anybody wants to step up as the PTL or just take over the development, it would be great and I could provide all the required help and guidance. In this case, I won’t be announcing my candidacy in the upcoming election. Thanks Renat Akhmerov @Tailify

3 5

How to get volume quota usage
by Braden, Albert 01 Feb '21

01 Feb '21

I'm auditing quota usage for all of our projects, and checking them all in Horizon would be a lot of work so I wrote a little script to pull them all and build a spreadsheet. I can get network and compute usage with "openstack quota list -detail" but when I try to get volume usage it demurs: (openstack) [root@chrnc-area51-build-01 openstack]# openstack quota list --detail --volume --project test Volume service doesn't provide detailed quota information I can see the volume quota usage in Horizon. How can I get it on the CLI? I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.

2 2

[cinder] kioxia ready for driver and connector reviews
by Zohar Mamedov 01 Feb '21

01 Feb '21

Hi Cinder team, The Kioxia Kumoscale NVMeOF driver is ready for reviews. Gate tests are passing and "KIOXIA CI" is leaving comments on review threads. Here is an example of last response it posted: https://review.opendev.org/c/openstack/cinder/+/765304/2#message-6ca11d3b3c… All changes that the CI responded to can be found here: http://104.254.65.37/ (each will also have a gerrit_event.json which will have a ['change']['url'] with a link to the review page) Part of this contribution is a major expansion to the nvmeof connector in os-brick. This was initially discussed in the PTG meetings, and mentioned in some follow up IRC meetings. The main change is adding (optional) support for mdraid replication of nvmeof volumes. Target connection sharing is also implemented (removing the need for a separate nvmeof target per volume.) Additionally, a concurrency / race condition issue in the existing nvmeof connector (nvmeof.py line 120) is resolved as part of these changes. Though this change is a big rework, it is in line with the spec, which had an in depth review and was approved: https://review.opendev.org/c/openstack/cinder-specs/+/766730 We are dedicated to addressing possible regressions, however, following a review of existing CIs using or testing the nvmeof connector, none were found to be using it. All suspected CIs seem to pass or be using a different connector. This is to the best of our knowledge, we suspect that Kioxia may be the first CI to be testing with NVMeOF protocol and driver. If we missed anything, please let us know with a review comment. Kioxia Kumoscale driver review: https://review.opendev.org/c/openstack/cinder/+/768574 NVMeOF connector review: https://review.opendev.org/c/openstack/os-brick/+/768575 Thank you! Zohar

1 0

[ironic] Reducing EM stable branch testing
by Julia Kreger 01 Feb '21

01 Feb '21

Greetings everyone, Last week, during our Wallaby mid-cycle[0] meeting, we discussed removing testing on older (extended, and approaching extended) maintenance branches. In part because of the number of integration jobs which we have and the overhead being so high to keep them operational at this point in time across every repository we use. The consensus we reached was to begin peeling back fragile and resource intensive jobs on these older branches. Mainly "multinode" and grenade type jobs where we risk running the test virtual machine out of memory. We may opt to remove additional older integration jobs and minimize the scenarios as seems relevant, but we will approach that when that time comes. Please let us know if you have any questions or concerns. Thanks, Julia [0]: https://etherpad.opendev.org/p/ironic-wallaby-midcycle

1 0

Failed to Install Ceilometer using devstack pike
by Nitish Goel 01 Feb '21

01 Feb '21

Hi , Please help.! I try to install heat ,ceilometer and aodh openstack using the devstack pike version. But getting the below error: *no matching distribution found for oslo.log>=4.3.0* pip version 9.x python version 2.7 Next Attempt: Upgraded the pip to 20.x but still gets the same error. Next Attempt: *Updated the python 3.6(pip with the old version) but then gets the below error:* Traceback (most recent call last): File "/opt/stack/devstack/tools/outfilter.py", line 85, in <module> sys.exit(main()) File "/opt/stack/devstack/tools/outfilter.py", line 53, in main outfile = open(opts.outfile, 'a', 0) ValueError: can't have unbuffered text I/O Next Attempt: Initially heat was installed successfully, but after the above errors from ceilometer and aodh, I commented them and executed stack.sh but now heat installation also fails with below error: *(pip with 9.x and python 2.7)* CRITICAL keystone [None req-ddd57555-0e66-4e94-8349-044e79573d01 None None] Unhandled error: TypeError: __init__() got an unexpected keyword argument 'encoding' 2021-02-01 11:43:37.083 | ERROR keystone Traceback (most recent call last): 2021-02-01 11:43:37.083 | ERROR keystone File "/usr/local/bin/keystone-manage", line 10, in <module> 2021-02-01 11:43:37.083 | ERROR keystone sys.exit(main()) 2021-02-01 11:43:37.083 | ERROR keystone File "/opt/stack/keystone/keystone/cmd/manage.py", line 45, in main 2021-02-01 11:43:37.083 | ERROR keystone cli.main(argv=sys.argv, config_files=config_files) 2021-02-01 11:43:37.083 | ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 1331, in main 2021-02-01 11:43:37.083 | ERROR keystone CONF.command.cmd_class.main() 2021-02-01 11:43:37.083 | ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 380, in main 2021-02-01 11:43:37.083 | ERROR keystone klass.do_bootstrap() 2021-02-01 11:43:37.083 | ERROR keystone File "/opt/stack/keystone/keystone/cmd/cli.py", line 190, in do_bootstrap 2021-02-01 11:43:37.083 | ERROR keystone domain=default_domain) 2021-02-01 11:43:37.083 | ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 110, in wrapped 2021-02-01 11:43:37.084 | ERROR keystone __ret_val = __f(*args, **kwargs) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/resource/core.py", line 723, in create_domain 2021-02-01 11:43:37.084 | ERROR keystone domain_id, project_from_domain, initiator) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/common/manager.py", line 110, in wrapped 2021-02-01 11:43:37.084 | ERROR keystone __ret_val = __f(*args, **kwargs) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/resource/core.py", line 221, in create_project 2021-02-01 11:43:37.084 | ERROR keystone ret['domain_id']) 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/dogpile/cache/region.py", line 1239, in set_ 2021-02-01 11:43:37.084 | ERROR keystone self.set(key, value) 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/dogpile/cache/region.py", line 983, in set 2021-02-01 11:43:37.084 | ERROR keystone key = self.key_mangler(key) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/common/cache/core.py", line 87, in key_mangler 2021-02-01 11:43:37.084 | ERROR keystone key = '%s:%s' % (key, invalidation_manager.region_id) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/common/cache/core.py", line 45, in region_id 2021-02-01 11:43:37.084 | ERROR keystone self._region_key, self._generate_new_id, expiration_time=-1) 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/dogpile/cache/region.py", line 833, in get_or_create 2021-02-01 11:43:37.084 | ERROR keystone async_creator) as value: 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/dogpile/lock.py", line 154, in __enter__ 2021-02-01 11:43:37.084 | ERROR keystone return self._enter() 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/dogpile/lock.py", line 87, in _enter 2021-02-01 11:43:37.084 | ERROR keystone value = value_fn() 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/dogpile/cache/region.py", line 788, in get_value 2021-02-01 11:43:37.084 | ERROR keystone value = self.backend.get(key) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/common/cache/_context_cache.py", line 72, in get 2021-02-01 11:43:37.084 | ERROR keystone value = self._get_local_cache(key) 2021-02-01 11:43:37.084 | ERROR keystone File "/opt/stack/keystone/keystone/common/cache/_context_cache.py", line 57, in _get_local_cache 2021-02-01 11:43:37.084 | ERROR keystone value = msgpackutils.loads(value) 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/oslo_serialization/msgpackutils.py", line 487, in loads 2021-02-01 11:43:37.084 | ERROR keystone return msgpack.unpackb(s, ext_hook=ext_hook, encoding='utf-8') 2021-02-01 11:43:37.084 | ERROR keystone File "/usr/local/lib/python2.7/dist-packages/msgpack/fallback.py", line 126, in unpackb 2021-02-01 11:43:37.084 | ERROR keystone unpacker = Unpacker(None, max_buffer_size=len(packed), **kwargs) 2021-02-01 11:43:37.084 | ERROR keystone TypeError: __init__() got an unexpected keyword argument 'encoding' 2021-02-01 11:43:37.084 | ERROR keystone Thanks, Nitish Goel

1 0