- openstack-discuss - lists.openstack.org

All VMs fail when --max exceeds available resources
by Albert Braden 20 Nov '19

20 Nov '19

I was experimenting in our dev cluster with CPU pinning and filters. After I was done I ran Ansible to put everything back like it was, but the scheduler is broken in 2 ways and I can't find the problem in my config. The first symptom is that if I use -max to create more VMs than the hypervisors can support, all of them go to ERROR. Before I changed things, --max would fill the hypervisors and only the extra VMs would go to ERROR. I'll email separately about the other one; this is already getting long. When I look at the logs, I see the "Starting to schedule" line with the list of instance UUIDs, and then "Attempting to claim resources" log entries. Hosts are selected; they are starting with 0 instances: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:56.880 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: ef1f7493-f792-4c3b-bf50-8e68b3d553ac] Selected host: (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 120696MB disk: 986112MB io_ops: 0 instances: 0 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346 Scheduler starting with the correct number of HV: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:56.888 1409571 DEBUG nova.filters [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70 I see the HV being weighed and the number of instances on each HV increasing from 0: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:57.106 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filtered [(us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: 85880MB disk: 880640MB io_ops: 1 instances: 2, (us01odc-dev1-hv001, us01odc-dev1-hv001.internal.synopsys.com) ram: 89976MB disk: 932864MB io_ops: 0 instances: 1, (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 89976MB disk: 929792MB io_ops: 1 instances: 1] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:435 us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:57.107 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Weighed [WeighedHost [host: (us01odc-dev1-hv001, us01odc-dev1-hv001.internal.synopsys.com) ram: 89976MB disk: 932864MB io_ops: 0 instances: 1, weight: 2.40576402895], WeighedHost [host: (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 89976MB disk: 929792MB io_ops: 1 instances: 1, weight: 1.11693447844], WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: 85880MB disk: 880640MB io_ops: 1 instances: 2, weight: 0.961725168625]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454 The number of instances goes beyond the HV capacity: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:57.602 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Weighed [WeighedHost [host: (us01odc-dev1-hv001, us01odc-dev1-hv001.internal.synopsys.com) ram: 59256MB disk: 876544MB io_ops: 1 instances: 2, weight: 1.17395901159], WeighedHost [host: (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 59256MB disk: 873472MB io_ops: 2 instances: 2, weight: 0.435549629144], WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: 55160MB disk: 824320MB io_ops: 2 instances: 3, weight: 0.292945361349]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454 Hosts are still being selected, but they are way over capacity already: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:58.696 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: 579dd4e2-d5d5-445f-905f-84cfd93146f6] Selected host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 711680MB io_ops: 4 instances: 5 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346 Then we start to see the warnings: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:58.817 1409571 WARNING nova.scheduler.client.report [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Unable to submit allocation for instance 5de08185-1ef2-4c92-8a19-5f09ec27be71 (409 {"errors": [{"status": 409, "request_id": "req-6cdd0b7a-bdbd-486f-9792-20840ea4a72e", "code": "placement.undefined_code", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'f20fa03d-18f4-486b-9b40-ceaaf52dabf8'. The requested amount would exceed the capacity. ", "title": "Conflict"}]}) And then they all fail and are deleted: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.057 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Unable to successfully claim against any host. _schedule /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:242 us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.058 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Cleaning up allocations for [u'ef1f7493-f792-4c3b-bf50-8e68b3d553ac', u'80afcfd7-ce25-4fc4-8b0d-b581a31b87bd', u'22cf9509-39e8-456a-b32c-e950cc597266', u'd6fc0b66-cd44-410d-96c9-590c22f1e21b', u'713a12fb-8bbf-4eac-ae02-68cb007fa34e', u'30c70c4d-1447-4481-bdc5-816835180ac6', u'be955f43-ac56-4531-ada8-16ce966211c7', u'b91820f6-775e-4abb-b28e-5b9b065819c2', u'83472954-2197-4e65-b7e7-1fe892f28458', u'd862bd74-494b-4c29-94ad-b80fc6c113e8', u'80a69dba-8011-403f-87fb-c39ef17ba467', u'c440d8d1-4da2-4c27-af9f-dd5afa19d083', u'5b301d69-a4f1-4393-8d67-4286c9873490', u'579dd4e2-d5d5-445f-905f-84cfd93146f6'] _cleanup_allocations /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:299 us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.124 1409571 INFO nova.scheduler.client.report [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Deleted allocation for instance ef1f7493-f792-4c3b-bf50-8e68b3d553ac us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.190 1409571 INFO nova.scheduler.client.report [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Deleted allocation for instance 80afcfd7-ce25-4fc4-8b0d-b581a31b87bd And the errors: us01odc-dev1-ctrl1:/var/log/nova/nova-conductor.log:2019-11-19 11:40:00.201 1801903 WARNING nova.scheduler.utils [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available. us01odc-dev1-ctrl1:/var/log/nova/nova-conductor.log:2019-11-19 11:40:00.205 1801903 WARNING nova.scheduler.utils [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: 5b301d69-a4f1-4393-8d67-4286c9873490] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

1 0

[tc] Recent updates and action items
by Jean-Philippe Evrard 20 Nov '19

20 Nov '19

Hello TC members, There are still business opportunities merging for 2019. The process for 2020 need to start soon. We have action items from the Summit and PTG [1] to take care. Please review etherpads and check if any part actions that you already been assigned or whould like to help. Here are a few action items already if you want to help: - ttx: will be working on the technical vision reflection, as we arrive at the end of the road (2019). - gmann: will be working on the multiple goals he has, including python versions. A little help would be welcomed I guess :) - mnaser: will be working on telemetry and an oslo project to expose metrics for prometheus. (If I didn't get that one wrong) - jungleboyj: will write a blog post about the analysis of the Foundation user survey. We should read it when it's ready and prepare the next one - mugsie: is writing a new proposal for release naming based on what we discussed at the summit - We all should think about stable policy and what to do about it, based on the feedback from TheJulia/smcginnis/release+requirements team. Maybe diablo_rojo can follow this? - everyone: we should promote the first contact sig to organisations that might need help on getting things achieved in OpenStack. - zaneb: will propose documentation of the process on how TC members should vote on conflicting resolutions when there is no predetermined consensus - mnaser: will be the point of contact for the infra matters of static hosting - ttx: will investigate a merge tc/uc - evrardjp: will start the new ideas concept that was floated before the summit and got traction during the summit. PowerVMStackers is now a SIG [2]. However, mnaser suggested to move to a multi-arch SIG. Mohammed, do you want to get started on this? V release naming process is now started [3]. [1] https://etherpad.openstack.org/p/PVG-TC-PTG [2] https://review.opendev.org/#/c/680438/ [3] https://review.opendev.org/#/c/693266/ Regards, Jean-Philippe & Rico

3 2

[release][stable][telemetry]Please add Rong Zhu to ceilometer-stable-maint group
by Rong Zhu 20 Nov '19

20 Nov '19

Hi Stable Maintenance Core team, I am the current Telemetry PTL, could you please add me to ceilometer-stable-maint group. And please also add Lingxian Kong to this group. -- Thanks, Rong Zhu

3 2

[glance] glance_store tests failed
by Naohiro Sameshima（鮫島直洋）(Group) 20 Nov '19

20 Nov '19

Hi, When I run a test with `tox -e py37` in glance_store, two tests failed. The command what I ran is below. 1. git clone https://opendev.org/openstack/glance_store.git 2. tox -e py37 Is there something wrong with how to run test? Thanks & Best Regards, ============================== Failed 2 tests - output below: ============================== glance_store.tests.unit.test_filesystem_store.TestStore.test_add_check_metadata_list_with_valid_mountpoint_locations -------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ b'Traceback (most recent call last):' b' File "/Users/sameshima/glance_store/glance_store/tests/unit/test_filesystem_store.py", line 215, in test_add_check_metadata_list_with_valid_mountpoint_locations' b' self.assertEqual(in_metadata[0], metadata)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual' b' self.assertThat(observed, matcher, message)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat' b' raise mismatch_error' b"testtools.matchers._impl.MismatchError: {'id': 'abcdefg', 'mountpoint': '/tmp'} != {}" b'' glance_store.tests.unit.test_multistore_filesystem.TestMultiStore.test_add_check_metadata_list_with_valid_mountpoint_locations ------------------------------------------------------------------------------------------------------------------------------ Captured traceback: ~~~~~~~~~~~~~~~~~~~ b'Traceback (most recent call last):' b' File "/Users/sameshima/glance_store/glance_store/tests/unit/test_multistore_filesystem.py", line 276, in test_add_check_metadata_list_with_valid_mountpoint_locations' b' self.assertEqual(in_metadata[0], metadata)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual' b' self.assertThat(observed, matcher, message)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat' b' raise mismatch_error' b"testtools.matchers._impl.MismatchError: {'id': 'abcdefg', 'mountpoint': '/tmp'} != {'store': 'file1'}" This email and all contents are subject to the following disclaimer: https://hello.global.ntt/en-us/email-disclaimer

3 3

[kolla][gnocchi] Issue on Gnocchi metricd when using Ceph as backend but with cache tier.
by Eddie Yen 20 Nov '19

20 Nov '19

Hi everyone, I'm not sure if it's good to ask at this place but still want to make sure that some one has a same issue as me. Currently we're using Kolla Openstack (R version), and Gnocchi version is 4.3.2 (ubuntu-source). The issue is, we always get below error trace in gnocchi-metricd after launched VM or uploaded images that let Ceilometer to create metrics in few environments. https://pastebin.com/fPZSFWaE After few investigations, we found that this only happens on the environment which enabled cache tier in Ceph. And also found the gnocchi cache pool only got 14 Bytes data inside, and no data inside the OSD pool one. We tried remove the cache tier from gnocchi pool then restart the gnocchi services. Then everything become normal. Can saw Gnocchi work normally to write data into Ceph pool. Does anyone had a same problem? Eddie.

1 0

[Watcher] irc meeting at 8:00 UTC today
by Canwei Li 20 Nov '19

20 Nov '19

Hi,Watcher team meeting is in the #openstack-meeting-alt channel. The agenda is available on https://wiki.openstack.irg/wiki/Watcher_Meeting_Agenda Thanks! Canwei Li 发自139邮箱

2 1

[all][tc] What happened in OpenStack Governance last month
by Jean-Philippe Evrard 20 Nov '19

20 Nov '19

Hello everyone, I hope you're all back from the summit to a normal life (nod to all the people at conferences, like KubeCon or devops days)! Here are a few things you might want to know about... First, for Python2.7 and Python3: We now removed references to Python 2.7 in the PTI (Project Testing Interface) [1]. In Ussuri we no longer require projects to test for python 2.7, so remove it from the list of requirements in the PTI. You can find current PTI for python in [2]. For py2 to py3 migration timing, there is now a ML [4] and etherpad[6] to discuss about specific schedule (now out in [6]), please join our discussions! We need each teams to pay attention on their python dependency and check if that schedule works. The release schedule will be update accordingly. We recently merged updates to the drop py2 goal too [9]. Then, for changes in projects, SIGs or adjacent communities: - If you didn't know: Airship is now confirmed as official project under OSF [3]. Congrats Airship team! - In OpenStack projects, we now have new barbican-ui, and manila- ganesha charms repositories. Microversion parse was adopted by the oslo team [7], and the API site was un-retired for redirect reasons. - Large scale SIG is under plan [8]. Please join, if you're interested on it. - PowerVM is now a SIG instead of a project. In the future, we might want to have a Multi-Arch SIG. - The Technical Writing SIG is born, replacing the openstack docs project. - Designate was promoted as a 2019 "business opportunity" for investment. We are now thinking for 2020. Thank you all for reading until this line! ;) [1] https://review.opendev.org/#/c/686582 [2] https://governance.openstack.org/tc/reference/pti/python.html [3] http://lists.openstack.org/pipermail/foundation/2019-October/002804.html [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.… [5] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2019… [6] https://etherpad.openstack.org/p/drop-python2-support [7]: https://review.opendev.org/#/c/689754/ [8] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010757… [9]: https://review.opendev.org/#/c/694161/ Regards, Jean-Philippe & Rico

1 0

On next minimum libvirt / QEMU versions for "V" release
by Kashyap Chamarthy 20 Nov '19

20 Nov '19

Heya, The last time we incremented versions for libvirt and QEMU was during the "Stein" release[1]. For "Train" we didn't do any. Although we advertized NEXT_MIN_{LIBVIRT,QEMU} versions for "Train" release to be libvirt 4.0.0 and QEMU 2.11.0, but we actually didn't bump; we'll do that for "Ussuri". But before we bump the versions for "Ussuri", we need to pick NEXT_MIN versions for the "V" release. Based on the updated the DistroSupportMatrix page[2], it looks we can pick the next libvirt and QEMU versions for "V" release to the following: libvirt: 5.0.0 [GAed on: 15-Jan-2019] QEMU: 4.0.0 [GAed on: 24-Apr-2019] I have the initial patch here[3] for comments. Debian, Fedora, Ubuntu[4], CentOS, RHEL currently already ship the above versions (actually, higher than those). And it is reasonable to assume -- but let's confirm below -- that openSUSE, SLES, and Oracle Linux would also have the above versions available by "V" release time. Action Items for Linux Distros ------------------------------ (a) Oracle Linux: Please update your libvirt/QEMU versions for Oracle Linux 8? I couldn't find anything related to libvirt/QEMU here: https://yum.oracle.com/oracle-linux-8.html. (My educated guess is: the versions roughly match what's in CentOS/RHEL.) (b) openSUSE and SLES: Same request as above. Andreas Jaegaer said on #openstack-infra that the proposed versions for 'V' release should be fine for SLES. (And by extension open SUSE, I assume.) - - - Assuming Oracle Linux and SLES confirm, please let us know if there are any objections if we pick NEXT_MIN_* versions for the OpenStack "V" release to be libvirt: 5.0.0 and QEMU: 4.0.0. Comments / alternative proposals welcome :-) [1] https://opendev.org/openstack/nova/commit/489b5f762e -- Pick next minimum libvirt / QEMU versions for "T" release, 2018-09-25) [2] https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix [3] https://review.opendev.org/694821 -- [RFC] Pick NEXT_MIN libvirt/QEMU versions for "V" release [4] For Ubuntu, I updated the versions based on what is in the Cloud Archive repo for "Bionic" (the LTS) release: http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/train_vers… -- /kashyap

2 3

[scientific-sig] Meeting CANCELLED - SC19
by Stig Telfer 20 Nov '19

20 Nov '19

Hi All - Unfortunately, we must cancel this week’s meeting as all the SIG chairs and many of the members are currently in Denver for Supercomputing 2019. Apologies, Stig

1 0

Stable branch tox now installed with Python 3 in gate Bionic nodes
by Ian Wienand 20 Nov '19

20 Nov '19

Hello, In the infra meeting we discussed that tox is installed with Python 3 on Bionic hosts in the gate now [1]. This can lead to tox jobs that do not specify a basepython running under python3. This was an unintentional change that came from [2] which uses the python version diskimage-builder thinks is the default on the platform to install tox. On Bionic this is Python 3. (The reasoning behind this was because of CentOS 8; where we are shipping a true Python3 only distribution) Opinion was mixed; this can be fixed by setting basepython=2 in the tox.ini. It's arguable this is the most correct solution and what should have been done, as it makes it agnostic to platform changes. This includes anyone who might be running *outside* the gate trying to replicate jobs. However, I'm sympathetic to the argument that things changing on stable jobs is very annoying too. We could put in a workaround to install tox under python2 on Bionic, restoring the old behaviour. Personally I'm of the opinion this just hides what is a real failure; that the test is only compatible with Python 2 but doesn't specify it. I feel like we should come down on the side of just fixing tests so they are isolated from this, be it in the gate or otherwise. Better to fix the gate quickly than have potential users struggling with running tests in incompatible environments, for mine. However, I'll take advice if the problems are just too big to fix piecemeal, and we need a workaround. Thanks, -i [1] http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-11-19-19.05.l… [2] https://review.opendev.org/#/c/686524/

1 0

[all][tc][forum] Summarizing Ussuri cycle community goal discussion in Forum and PTG
by Ghanshyam Mann 19 Nov '19

19 Nov '19

Hello Everyone, We discussed Ussuri goals in Forum as well as in PTG. I would like to summarize the discussions here. * Forum[1]: Three goals were discussed in details: 1. Drop Python 2.7 Support We discussed keeping the CI/CD support for Swift which is the only project keeping the py2 support. Swift needs the devstack to keep installing on py2 env with the rest of the services on py3 (same as old jobs when Swift was on py2 by default on devstack). There is no oslo dependency from swift and all the other dependency will be capped for py2 version. Requirements check job currently checks that if openstack/requirements list two entries for a requirement (one for <=2.7 and one for >) that the repo under test also has both entries. smcginnis already pushed the changes[2] to handle dual python version requirements. Everything else will go as discussed in ML[3] and this is already accepted as Ussuri goal[4]. 2. Project Specific New Contributor & PTL Docs - As per feedback in Forum sessions, this is a good goal which will make documentation more consistent. All the projects should edit their contributor.rst to follow a more complete template and adjust/add PTL documentation. - This is accepted as a pre-approved as Ussuri goal - Kim Hindhart is working on getting EU funding for people to work on OpenStack and they like consistent documentation. - diablo_rojo already updated the goal proposal patch and it is up to get wider feedback. 3. Switch remaining legacy jobs to Zuul v3 and drop legacy support - Grenade job is not yet on zuulv3 which is required to finish first. - Few projects waiting for big projects finishing the zuulv3 migration first. - This needs more work and can be a "pre-approved" thing for V, and this would be split to focus on the Grenade work in U. Other than above 3 goals, there were few more ideas for goal candidate and good to go in goal backlogs etherpad: - cdent: stop using paste, pastedeploy and WSME Note from Chris: This does not need to be a community goal as such but requires the common solution from TC WSME is still used, has contributions, and at least a core or two - cmurphy: Consistent and secure default policies https://etherpad.openstack.org/p/PVG-keystone-forum-policy Going with pop-up team first - support matrix documentation to be consistent across projects. going with pop-up team (fungi can propose the pop-up team in governance) first Richard Pioso (rpioso) to help fungi on this once consistent framework is identified, the pop-up team can expire with the approval of a related cycle goal for implementing it across remaining projects * PTG[5]: We all agreed to select the Ussuri goal asap because the Ussuri cycle already started and projects are waiting for the final goals. TC discussed this further in PTG and agreed on below two goals to be selected as the Ussuri goal. 1. Drop Python 2.7 Support - Already Accepted. 2. Project Specific New Contributor & PTL Docs - Under Review Goal "Switch remaining legacy jobs to Zuul v3 and drop legacy support" will be pre-selected for V cycle, that does not mean to stop the review on proposed goal or any ongoing work. All ongoing efforts will continue on this. [1] https://etherpad.openstack.org/p/PVG-ussuri-goal-forum [2] https://review.opendev.org/#/c/693631/ [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010371.… [4] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html [5] L76 https://etherpad.openstack.org/p/PVG-TC-PTG -gmann

1 0

[magnum] Kubernetes cluster issue
by Namrata Sitlani 19 Nov '19

19 Nov '19

Hello Folks, >From Thursday last week (Nov 14), Magnum is unable to spin up working Kubernetes clusters. We run on Rocky Openstack release. All our Kubernetes pods show CrashLoopBackOff status. We use the following commands to create the Kubernetes clusters : http://paste.openstack.org/show/786348/ . The deployment fails with the following output : http://paste.openstack.org/show/786287/. We tried deploying Kubernetes v1.13.12, v1.14.8, v1.15.5 and v1.16.2 without success. However, if we use version v1.14.6 we can successfully deploy our clusters. Unfortunately, we cannot use v1.14.6 in production because it is not patched for the CVE-2019-11253 vulnerability . Since this stopped working for us on Thursday, we think that the image update https://hub.docker.com/r/openstackmagnum/kubernetes-apiserver/tags done 6 days ago is the culprit. (We previously deployed clusters with versions v1.15.5 and v1.14.8 successfully.) Can you please confirm our findings and help us find a way forward? We can provide more logs if needed. Thank you very much, Namrata Sitlani

2 1

[RabbitMQ][cinder] Listen to messages
by Blom, Merlin, NMU-OI 19 Nov '19

19 Nov '19

Hey there, it seems to me as if ask.openstack.org is down, so I ask my question here: I'd like to listen to oslo messages from cinder as I do for neutron and octavia to know what is going on. For me the following code worked for neutron: EXCHANGE_NAME = os.getenv('EXCHANGE_NAME', 'neutron') ROUTING_KEY = os.getenv('ROUTING_KEY', 'notifications.info') QUEUE_NAME = os.getenv('QUEUE_NAME', 'messaging_queue') BROKER_URI = os.getenv('BROKER_URI', 'UNDEFINED') BROKER_PASSWORD = os.getenv('BROKER_PASSWORD', '') class Messages(ConsumerMixin): def __init__(self, connection): self.connection = connection return def get_consumers(self, consumer, channel): exchange = Exchange(EXCHANGE_NAME, type="topic", durable=False) queue = Queue(QUEUE_NAME, exchange, routing_key=ROUTING_KEY, durable=False, auto_delete=True, no_ack=True) return [consumer(queues=[queue], callbacks=[self.on_message])] def on_message(self, body, message): try: print(message) except Exception as e: log.info(repr(e)) if __name__ == "__main__": log.info("Connecting to broker {}".format(BROKER_URI)) with BrokerConnection(hostname=BROKER_URI, userid='messaging', password=BROKER_PASSWORD, virtual_host='/'+EXCHANGE_NAME, heartbeat=4, failover_strategy='round-robin') as connection: Messaging(connection).run() BrokerConnection.connection.close() But on the cinder vhost (/cinder) I can't find an exchange that the code is working on. (cinder, cinder-backup, .) I tried using the rabbitmq tracer: <https://www.rabbitmq.com/firehose.html> https://www.rabbitmq.com/firehose.html And got all the cinder messages but I don't want to use it in production because of performance issues. Does anyone have an idea how to find the correct exchange for the notification info queue in cinder? Cheers, Merlin

2 2

Trove Image issue
by rabia.shaheen＠xflowresearch.com 19 Nov '19

19 Nov '19

Hi Team, I have deployed kolla-ansible(Stein) with Trove enable and using Trove prebuild images to build the Database VM but VM is constantly stuck in build state. I am not sure how to use prebuild image key which is in https://opendev.org/openstack/trove/src/branch/master/integration/scripts/fi les/keys folder. Can you please guide me regarding the key usage for prebuild images (http://tarballs.openstack.org/trove/images/) To build my own trove image on kolla-ansible, is there any specific guide available for it? Warm Regards, Rabia Shaheen Lead Engineer, xFlow Research Inc. +923075462720 (GMT+5) <mailto:rabia.shaheen@xflowresearch.com> rabia.shaheen(a)xflowresearch.com <http://www.xflowresearch.com/> www.xflowresearch.com

2 1

how to update back-end storage online
by yj x 19 Nov '19

19 Nov '19

Hi, I want to implement a qemu block driver which like qeme/block/iscsi.c. Here are my questions: some targets have be attached to guest system, and then the targets info have changed. How can I update the connection to targets online? I mean the disk in guest system does not change, just update the connection to the back-end storage, and the operation can't affect guest system. I can't find a nova-api or libvirt-api to meet my needs now. Does anyone help me? Thanks.

1 0

[keystone] post-PTG virtual meeting reminder
by Colleen Murphy 18 Nov '19

18 Nov '19

Hi keystoners, As a reminder, we'll be holding our post-PTG meeting tomorrow at 14:00 UTC (with the daylight savings time change that makes it 6:00 PST (RIP me) / 9:00 EST / 19:30 IST). Last week I briefly floated the idea of rescheduling it but quickly decided there was not enough notice, so we're holding it at the day and time that we've had planned for the last few weeks. Our agenda and notes: https://etherpad.openstack.org/p/keystone-shanghai-ptg We'll try jitsi.org again, I think the technical issues we were having last time were on my end and not related to the platform: https://meet.jit.si/keystone-ptg Please please please review the roadmap board and assign yourself to items you have been working on and update their status: https://tree.taiga.io/project/keystone-ussuri-roadmap/kanban . We'll be going over the board together tomorrow, and updating it ahead of time will save us time as a group. Colleen

1 0

[manila] No IRC meeting on Nov 7 and Nov 21 2019
by Goutham Pacha Ravi 18 Nov '19

18 Nov '19

Hello Zorillas and interested stackers, Due to a part of our community attending the Open Infrastructure Summit+PTG (Nov 4-8, 2019) and KubeCon+CloudNativeCon (Nov 18-21, 2019), I propose that we cancel the weekly IRC meetings on Nov 7th and Nov 21st. If you'd like to discuss anything during these weeks, please chime in on freenode/#openstack-manila, or post to this mailing list. Thanks, Goutham

1 1

Shanghai PTG Team Photos
by Kendall Waters 18 Nov '19

18 Nov '19

Hi everyone, Thank you for attending the Project Teams Gathering in Shanghai! If your team took a team picture, you can find a copy of the photo file in this Dropbox folder: https://www.dropbox.com/sh/1my6wdtuc1hf58o/AACU49pjWxzFNzcZJgjLG8n1a?dl=0 If you are unable to open Dropbox, please send me an email with which team photo you are looking for and I can send you the file directly. Cheers, Kendall Kendall Waters OpenStack Marketing & Events kendall(a)openstack.org

1 0

[Openstack] FOSDEM 2020 Virtualization & IaaS Devroom CfP
by Piotr Kliczewski 18 Nov '19

18 Nov '19

Friendly reminder that there are 2 weeks before the submission deadline. Room day update: This year Virt and IaaS room will be on the 2nd of February. See you all at FOSDEM!

1 0

Scheduler sends VM to HV that lacks resources
by Albert Braden 18 Nov '19

18 Nov '19

If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU." https://paste.fedoraproject.org/paste/6N3wcDzlbNQgj6hRApHiDQ I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU." https://paste.fedoraproject.org/paste/lGlVpfbB9C19mMzrWQcHCQ I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard: enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full: https://paste.fedoraproject.org/paste/6SX9pQ4V1KnWfQkVnfoHOw This is the command line I used: openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB

5 14

[neutron][drivers] Drivers meeting cancel
by Slawek Kaplonski 18 Nov '19

18 Nov '19

Hi, I can't attend next drivers meeting on Friday, 22.11.2019. I know also that 2 other members of drivers team will not be able to be on the meeting so as we will not have quorum on this meeting, lets cancel it. See You on the meeting next week, on 29.11.2019 where (I hope) we should have quorum even if it's just after Thanksgiving. -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[cloudkitty] 18/11 IRC meeting recap
by Luka Peschke 18 Nov '19

18 Nov '19

Hello, This is the recap for today's IRC meeting of the cloudkitty team. The agenda can be found at [1] and the logs can be found at [2]. CloudKitty 11.0.1 ================= CloudKitty 11.0.1 has been released. It includes a fix for security issue on the GET /v1/dataframes and GET /v2/dataframes endpoints which had been introduced during the train development cycle. Various updates =============== Two patches had been discussed during the previous meeting: one for developer documentation for scope fetchers, and one allowing to group results by timestamps on GET /v2/summary. Both have been merged. API improvements ================ Two patches improving the API are currently under review: - The first one updates the way oslo.context and oslo.policy are used. External reviews on this one would be very helpful [3] - The second one improves the way various drivers are loaded in the v2 API. It is available at [4] Standalone cloudkitty dashboard =============================== Julien Pinchelimouroux (julien-pinchelim) has been working on a standalone dashboard (which will also support keystone authentication)for cloudkitty. A 0.1.0 release should happen during in Q4 2019. Some screenshots are available at [5] Cheers, -- Luka Peschke (peschk_l) [1] https://etherpad.openstack.org/p/cloudkitty-meeting-topics [2] http://eavesdrop.openstack.org/meetings/cloudkitty/2019/cloudkitty.2019-11-… [3] https://review.opendev.org/#/c/692333/ [4] https://review.opendev.org/#/c/686393/ [5] https://kutt.it/khA6yF

1 0

[horizon] Changing the release model to cycle-with-intermediary
by Akihiro Motoki 18 Nov '19

18 Nov '19

Hi, I just proposed a patch to change the horizon release model to cycle-with-intermediary. https://review.opendev.org/#/c/694772/ It was discussed during the Shanghai PTG. Horizon provides GUI to users. On the other hand, it is a library for horizon plugins. When horizon plugins would like to use recent changes or to avoid bugs, they need to consume beta releases of horizon. More frequent releases of horizon would make more sense, so I am proposing the release model change. If there are concerns, reply to this thread or drop comments in the review mentioned above. Thanks, Akihiro Motoki (irc: amotoki)

1 0

[neutron] Removing neutron-interconnection out of stadium
by Slawek Kaplonski 16 Nov '19

16 Nov '19

Hi neutrinos, As we discussed during the Shanghai PTG I just proposed changes to move neutron-interconnection project out of stadium to "x/" namespace. So it will not be official neutron project anymore after those changes. Patches for that are in [1] and [2] I also proposed to remove neutron-interconnection api-ref from neutron-lib. Patch is here [3]. Please review it :) [1] https://review.opendev.org/#/c/694478/ [2] https://review.opendev.org/#/c/694480/ [3] https://review.opendev.org/#/c/694466/ -- Slawek Kaplonski Senior software engineer Red Hat

4 5

[oslo][i18n][pbr] get_available_languages() always only returns ['en_US']
by info＠dantalion.nl 16 Nov '19

16 Nov '19

Hello, Across several projects I have noticed that both with unit tests or while the service is running calling olso_i18n.get_available_languages() only returns ['en_US'] (which is inserted as default). Even though the projects I have tested this on have several languages available in the locale directory. Calling any of the python setup.py extract_messages / compile_catalog / update_catalog / install commands does not solve this. However, when I manually copy the .mo files into /usr/share/locale/**/LC_MESSAGES it works as expected. My questions are: Surely there must be a less manual method to properly install all locale files but what is it? Why aren't the locale files installed by pbr when python setup.py install is called? I hope anyone knows the answers to these questions. Kind regards, Corne Lukken (Dantali0n)

1 0