October 2019 - openstack-discuss - lists.openstack.org

Re: [openstack-dev] Are we ready to put stable/ocata into extended maintenance mode?
by Alex Schultz 03 Mar '21

03 Mar '21

On Tue, Sep 18, 2018 at 1:30 PM Alex Schultz <aschultz(a)redhat.com> wrote: > > On Tue, Sep 18, 2018 at 1:27 PM, Matt Riedemann <mriedemos(a)gmail.com> wrote: > > The release page says Ocata is planned to go into extended maintenance mode > > on Aug 27 [1]. There really isn't much to this except it means we don't do > > releases for Ocata anymore [2]. There is a caveat that project teams that do > > not wish to maintain stable/ocata after this point can immediately end of > > life the branch for their project [3]. We can still run CI using tags, e.g. > > if keystone goes ocata-eol, devstack on stable/ocata can still continue to > > install from stable/ocata for nova and the ocata-eol tag for keystone. > > Having said that, if there is no undue burden on the project team keeping > > the lights on for stable/ocata, I would recommend not tagging the > > stable/ocata branch end of life at this point. > > > > So, questions that need answering are: > > > > 1. Should we cut a final release for projects with stable/ocata branches > > before going into extended maintenance mode? I tend to think "yes" to flush > > the queue of backports. In fact, [3] doesn't mention it, but the resolution > > said we'd tag the branch [4] to indicate it has entered the EM phase. > > > > 2. Are there any projects that would want to skip EM and go directly to EOL > > (yes this feels like a Monopoly question)? > > > > I believe TripleO would like to EOL instead of EM for Ocata as > indicated by the thead > http://lists.openstack.org/pipermail/openstack-dev/2018-September/134671.ht… > Bringing this backup to see what we need to do to get the stable/ocata branches ended for the TripleO projects. I'm bringing this up because we have https://review.openstack.org/#/c/647009/ which is for the upcoming rename but CI is broken and we have no interest in continue to keep the stable/ocata branches alive (or fix ci for them). Thanks, -Alex > Thanks, > -Alex > > > [1] https://releases.openstack.org/ > > [2] > > https://docs.openstack.org/project-team-guide/stable-branches.html#maintena… > > [3] > > https://docs.openstack.org/project-team-guide/stable-branches.html#extended… > > [4] > > https://governance.openstack.org/tc/resolutions/20180301-stable-branch-eol.… > > > > -- > > > > Thanks, > > > > Matt > > > > __________________________________________________________________________ > > OpenStack Development Mailing List (not for usage questions) > > Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

5 11

[Syntribos] Does anyone still use this?
by Gage Hugo 07 May '20

07 May '20

The Security SIG has recently been looking into updating several sites/documentation and one part we discussed the last couple meetings was the security tools listings[0]. One of the projects listed there, Syntribos, doesn't appear to have much activity[1] outside of zuul/python/docs maintenance and the idea of retiring the project was mentioned. However, there does seem to be an occasional bug-fix submitted, so we are sending this email out to see if anyone is still utilizing Syntribos. If so, please either respond here, reach out to us in the #openstack-security irc channel, or fill out the section in the Security SIG Agenda etherpad[2]. Thanks! [0] https://security.openstack.org/#security-tool-development [1] https://review.opendev.org/#/q/project:openstack/syntribos [2] https://etherpad.openstack.org/p/security-agenda

2 2

[openstack-ansible] dropping suse support
by Mohammed Naser 05 May '20

05 May '20

Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct… [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-funct…

6 9

[ops] nova wsgi config
by Arnaud Morin 27 Feb '20

27 Feb '20

Hey all, I am trying to configure apache as a WSGI. Is there any other documentation than this: https://docs.openstack.org/nova/stein/user/wsgi.html Is there any recommendations? Thanks in advance! -- Arnaud Morin

4 7

[cinder] consistency group not working
by Francois Scheurer 16 Jan '20

16 Jan '20

Dear Cinder Experts We are running the rocky release. |We can create a consistency group: openstack consistency group create --volume-type b9f67298-cf68-4cb2-bed2-c806c5f83487 fsc-consgroup Bug 1: but adding volumes is not working: openstack consistency group add volume c3f49ef0-601e-4558-a75a-9b758304ce3b b48752e3-641f-4a49-a892-6cb54ab6b74d c0022411-59a4-4c7c-9474-c7ea8ccc7691 0f4c6493-dbe2-4f75-8e37-5541a267e3f2 => Invalid volume: Volume is not local to this node. (HTTP 400) (Request-ID: req-7f67934a-5835-40ef-b25c-12591fd79f85) Bug 2: deleting consistency group is also not working (silently failing): openstack consistency group delete c3f49ef0-601e-4558-a75a-9b758304ce3b |||=> AttributeError: 'RBDDriver' object has no attribute 'delete_consistencygroup'| See details below. Using the --force option makes no difference and the consistency group is not deleted. Do you think this is a bug or a configuration issue? Thank you in advance. | Cheers Francois |Details: ==> /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-api-access.log <== 10.0.129.17 - - [27/Sep/2019:12:16:24 +0200] "POST /v3/f099965b37ac41489e9cac8c9d208711/consistencygroups/3706bbab-e2df-4507-9168-08ef811e452c/delete HTTP/1.1" 202 - 109720 "-" "python-cinderclient" ==> /var/lib/docker/volumes/kolla_logs/_data/cinder/cinder-volume.log <== 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server [req-9010336e-d569-47ad-84e2-8dd8b729939c b141574ee71f49a0b53a05ae968576c5 f099965b37ac41489e9cac8c9d208711 - default default] Exception during message handling: AttributeError: 'RBDDriver' object has no attribute 'delete_consistencygroup' 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/server.py", line 163, in _process_incoming 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 265, in dispatch 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/osprofiler/profiler.py", line 159, in wrapper 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server result = f(*args, **kwargs) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 3397, in delete_group 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server vol_obj.save() 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server self.force_reraise() 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/local/lib/python2.7/site-packages/cinder/volume/manager.py", line 3362, in delete_group 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server self.driver.delete_consistencygroup(context, cg, 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server AttributeError: 'RBDDriver' object has no attribute 'delete_consistencygroup' 2019-09-27 12:16:24.491 30 ERROR oslo_messaging.rpc.server| -- EveryWare AG François Scheurer Senior Systems Engineer Zurlindenstrasse 52a CH-8003 Zürich tel: +41 44 466 60 00 fax: +41 44 466 60 10 mail: francois.scheurer(a)everyware.ch web: http://www.everyware.ch

2 2

[tc][all] Updates on Ussuri cycle community-wide goals
by Ghanshyam Mann 16 Jan '20

16 Jan '20

Hello Everyone, We have two goals with their champions ready for review. Please review and provide your feedback on Gerrit. 1. Add goal for project specific PTL and contributor guides - Kendall Nelson - https://review.opendev.org/#/c/691737/ 2. Propose a new goal to migrate all legacy zuul jobs - Luigi Toscano - https://review.opendev.org/#/c/691278/ We are still looking for the Champion volunteer for RBAC goal[1]. If you have any new ideas for goal, do not hesitate to add in etherpad[2] [1] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010291.… [2] https://etherpad.openstack.org/p/PVG-u-series-goals -gmann & diablo_rojo

1 2

State of the Gate
by Matt Riedemann 21 Dec '19

21 Dec '19

Things are great! Surprise! I just wanted to let everyone know. Later! . . . . . Now that you've been tricked, on Halloween no less, I'm here to tell you that things suck right now. This is your periodic digest of issues. Grab some fun-sized candy bars and read on. I think right now we have three major issues. 1. http://status.openstack.org/elastic-recheck/index.html#1763070 This has resurfaced and I'm not sure why, nor do I think we ever had a great handle on what is causing this or how to work around it so if anyone has new ideas please chip in. 2. http://status.openstack.org/elastic-recheck/index.html#1844929 I've done some digging into this one and my notes are in the bug report. It mostly affects grenade jobs but is not entirely restricted to grenade jobs. It's also mostly on OVH and FortNebula nodes but not totally. From looking at the mysql logs in the grenade jobs, mysqld is (re)started three times. I think (1) for initial package install, (2) for stacking devstack on the old side, and (3) for stacking devstack on the new side. After the last restart, there are a lot of aborted connection messages in the msyql error log. It's around then that grenade is running post-upgrade smoke tests to create a server and the nova-scheduler times out communicating with the nova_cell1 database. I have a few patches up to grenade/devstack [1] to try some things and get more msyql logs but so far they aren't really helpful. We need someone with some more mysql debugging experience to help here, maybe zzzeek or mordred? 3. CirrOS guest SSH issues There are several (some might be duplicates): http://status.openstack.org/elastic-recheck/index.html#1848078 http://status.openstack.org/elastic-recheck/index.html#1808010 (most hits) http://status.openstack.org/elastic-recheck/index.html#1463631 http://status.openstack.org/elastic-recheck/index.html#1849857 http://status.openstack.org/elastic-recheck/index.html#1737039 http://status.openstack.org/elastic-recheck/index.html#1840355 http://status.openstack.org/elastic-recheck/index.html#1843610 A few notes here. a) We're still using CirrOS 0.4.0 since Stein: https://review.opendev.org/#/c/521825/ And that image was published nearly 2 years ago and there are no newer versions on the CirrOS download site so we can't try a newer image to see if that fixes things. b) Some of the issues above are related to running out of disk in the guest. I'm not sure what is causing that, but I have posted a devstack patch that is related: https://review.opendev.org/#/c/690991 tl;dr before Stein the tempest flavors we used had disk=0 so nova would fit the root disk to the size of the image. Since Stein the tempest flavors specify root disk size (1GiB for the CirrOS images). My patch pads an extra 1GiB to the root disk on the tempest flavors. One side effect of that is the volumes tempest creates will go from 1GiB to 2GiB which could be a problem if a lot of tempest volume tests run at the same time, though we do have a volume group size of 24GB in gate runs so I think we're OK for now. I'm not sure my patch would help, but it's an idea. As for the other key generation and dhcp lease failures, I don't know what to do about those. We need more eyes on these issues to generate some ideas or see if we're doing something wrong in our tests, e.g. generating too much data for the config drive? Not using config drive in some cases? Metadata API server is too slow (note we cache the metadata since [2])? Other ideas on injecting logs for debug? [1] https://review.opendev.org/#/q/topic:bug/1844929+status:open [2] https://review.opendev.org/#/q/I9082be077b59acd3a39910fa64e29147cb5c2dd7 -- Thanks, Matt

7 23

[resource-management-sig] Status of the "Resource Management" SIG
by Thierry Carrez 19 Dec '19

19 Dec '19

Hi, I was wondering about the status of the Resource Management SIG... It's been "forming" according to https://wiki.openstack.org/wiki/Res_Mgmt_SIG since January 2018... And I could'nt find a reference or log to any meeting after that. Does anyone have updated status on this one? Should it be removed from the list of active SIGs at https://governance.openstack.org/sigs/ ? -- Thierry Carrez (ttx)

5 6

[nova][ptg] Continue QoS port support
by Balázs Gibizer 15 Dec '19

15 Dec '19

Hi Novas! Here is my summarized plans for Ussuri about continuing the work on supporting qos neutron ports (those that has bandwidth resource request) in nova. What is missing: * Tempest coverage is missing for migrate and resize support that is merged in Train. This work is already underway and bugs has been caught [1][2] * Support evacuate, live migrate, unshelve. The work is described in [3][4] and the first set of patches for the evacuation support is up for review [5] * Support for cross cell resize with qos port needs some work. Matt prepared the cross cell resize code already in a way that no new RPC change will be needed [6] and I have a plan what to do [7]. * InstancePCIRequest persists parent_ifname during migration but the such change is not rolled back if the migration fails. This is ugly but I think it does not cause any issues [8]. I will look into this to remove the ugliness. The bandwidth support for the nova-manage heal_allocation tool was merged in Train. Originally I planned to backport that to Stein but that patch grown so big and incorporated may refactors along the way that I'm not sure any more that it is reasonable to backport it. I'm now thinking about keeping it as-is and suggesting operators to install Train nova in a virtualenv to run heal allocations for bandwidth aware servers if needed in Stein. I do have to run some manual tests to see if it actually works. Any feedback is welcome! cheers, gibi [1] https://bugs.launchpad.net/nova/+bug/1849695 [2] https://bugs.launchpad.net/nova/+bug/1849657 [3] https://blueprints.launchpad.net/nova/+spec/support-move-ops-with-qos-ports… [4] https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/supp… [5] https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:ma… [6] https://review.opendev.org/#/c/635080/43/nova/compute/manager.py@5375 [7] https://review.opendev.org/#/c/633293/49/nova/compute/manager.py@4742 [8] https://review.opendev.org/#/c/688387/6/nova/compute/manager.py@3404

2 3

[nova][cinder][ops] question/confirmation of legacy vol attachment migration
by Matt Riedemann 06 Dec '19

06 Dec '19

Hello Cinderinos, I've now got a working patch that migrates legacy volume attachments to new style v3 attachments [1]. The fun stuff showing it working is in this paste [2]. We want to do this data migration in nova because we still have a lot of compatibility code since Queens for pre-v3 style attachments and we can't remove that compatibility code (ever) if we don't first make sure we provide a data migration routine for operators to roll through. So for example if this lands in Ussuri we can can enforce a nova-status upgrade check in V and rip out code in X. Without digging into the patch, this is the flow: 1. On nova-compute restart, query the nova DB for instances on the compute host with legacy volume attachments. 2. For each of those, create a new style attachment with the host connector and update the BlockDeviceMapping information in the nova DB (attachment_id and connection_info). 3. Delete the existing legacy attachment so when the server is deleted the volume status goes back to 'available' due to proper attachment reference counting in the Cinder DB. My main question is on #3. Right now I'm calling the v3 attachment delete API rather than the v2 os-terminate_connection API. Is that sufficient to cleanup the legacy attachment on the storage backend even though the connection was created via os-initialize_connection originally? Looking at the cinder code, attachment_delete hits the connection terminate code under the covers [3]. So that looks OK. The only thing I can really think of is if a host connector is not provided or tracked with the legacy attachment, is that going to cause problems? Note that I think volume drivers are already required to deal with that today anyway because of the "local delete" scenario in the compute API where the compute host that the server is on is down and thus we don't have a host connector to provide to Cinder to terminate the connection. So Cinder people, are you OK with this flow? Hello Novaheads, Do you have any issues with the above? Note the migration routine is threaded out on compute start so it doesn't block, similar to the ironic flavor data migration introduced in Pike. One question I have is if we should add a config option for this so operators can enable/disable it as needed. Note that this requires nova to be configured with a service user that has the admin role to do this stuff in cinder since we don't have a user token, similar to nova doing things with neutron ports without a user token. Testing this with devstack requires [4]. By default [cinder]/auth_type is None and not required so by default this migration routine is not going to run so maybe that is sufficient? Hello Operatorites, Do you have any issues with what's proposed above? [1] https://review.opendev.org/#/c/549130/ [2] http://paste.openstack.org/show/781063/ [3] https://github.com/openstack/cinder/blob/410791580ef60ddb03104bf20766859ed9… [4] https://review.opendev.org/#/c/685488/ -- Thanks, Matt

5 10

[manila] Remote-only PTG poll
by Goutham Pacha Ravi 04 Dec '19

04 Dec '19

Hey Zorillas* and interested Stackers, (* manila contributors in the wild) A part of our community will be attending the Summit+Forum+PTG in Shanghai in a couple of weeks. However, we haven't figured out a way to enable remote participation during our PTG (Wednesday, 6th Nov 2019 between 9:00 AM and 4:30 PM) there. So, as discussed during our weekly meeting, I'd like to take a poll of when you would like to gather for a remote PTG that would supplement the gathering in Shanghai. Please take this poll and indicate your availability: https://xoyondo.com/dp/qQbsoHtNhi4DCki The prospective date for this meeting is between 2nd Dec and 13th Dec (Ussuri-1 Milestone is on 12th Dec). The meeting time will be 15:00 UTC to 22:00 UTC. This time may not work for our contributors in the Far East (APAC), but I'm hoping we'll address them primarily at the Shanghai get together, and relay their thoughts here. We will also be recording the entire meeting and will share notes on these lists. Thanks, Goutham

1 1

[oslo] Adoption of microversion-parse
by Thierry Carrez 27 Nov '19

27 Nov '19

Hi Osloites, A recent review of the OpenStack namespace yielded a library that no project team owns, and yet is specific to openstack and present in global-requirements: https://opendev.org/openstack/microversion-parse Ideally it should be adopted by one of our project teams. After discussing it with cdent, and since it is openstack-specific and depended on by most openstack projects, it feels like Oslo would be a good home for it. It would follow an "independent" release model and I don't think it would trigger much updates or maintenance. I can post patches to make it happen if you confirm you're OK to host it. Just keep me posted :) Cheers, -- Thierry Carrez (ttx)

7 13

Re: [ironic]: Timeout reached while waiting for callback for node
by fsbiz＠yahoo.com 26 Nov '19

26 Nov '19

Thanks Julia. In addition to what you mentioned this particular issue seems to have cropped up when we added 100 more baremetal nodes. I've also narrowed down the issue (TFTP timeouts) when 3-4 baremetal nodes are in "deploy" state and downloading the OS via iSCSI. Each iSCSI transfer takes about 6 Gbps and thus with four transfers we are over our 20Gbps capacity of the leaf-spine links. We are slowly migrating to iPXE so it should help. That being said is there a document on large scale ironic design architectures?We are looking into a DC design (primarily for baremetals) for upto 2500 nodes. thanks,Fred, On Wednesday, October 23, 2019, 03:19:41 PM PDT, Julia Kreger <juliaashleykreger(a)gmail.com> wrote: Greetings Fred! Reply in-line. On Tue, Oct 22, 2019 at 12:47 PM fsbiz(a)yahoo.com <fsbiz(a)yahoo.com> wrote: [trim] TFTP logs: shows TFTP client timed out (weird). Any pointers here? Sadly this is one of those things that comes with using TFTP. Issues like this is why the community tends to recommend using ipxe.efi to chainload as you can perform transport over TCP as opposed to UDP where in something might happen mid-transport. tftpd shows ramdisk_deployed completed. Then, it reports that the client timed out. Grub does tend to be very abrupt and not wrap up very final actions. I suspect it may just never be sending the ack back and the transfer may be completing. I'm afraid this is one of those things you really need to see on the console what is going on. My guess would be that your deploy_ramdisk lost a packet in transfer or that it was corrupted in transport. It would be interesting to know if the network card stack is performing checksum validation, but for IPv4 it is optional. [trim] This has me stumped here. This exact failure seems to be happening 3 to 4 times a week on different nodes.Any pointers appreciated. thanks,Fred.

3 3

[keystone] Upcoming team meetings and PTGs
by Colleen Murphy 26 Nov '19

26 Nov '19

Hi team, Tomorrow, October 29, we are having our virtual pre-PTG meeting from 14:00-17:30 UTC in lieu of our regular 16:00 IRC meeting. Tuesday, November 5 some of us will be in Shanghai for the Forum and I will not be available to chair the meeting, so I propose we skip it. Tuesday, November 12, we will have our virtual post-PTG meeting from 15:00-17:30 UTC in lieu of our regular 16:00 IRC meeting. Details about the PTG sessions can be found on the planning etherpad: https://etherpad.openstack.org/p/keystone-shanghai-ptg Colleen

1 2

[requirements][taskflow] networkx bump is blocked on what looks to be a taskflow issue.
by Matthew Thode 25 Nov '19

25 Nov '19

I've made a test review so people can see and test against, but it looks like the bump from 2.3 to 2.4 is causing some issues. hits nova,neutron,octavia,glance,cinder -- Matthew Thode

4 3

[all][tc] Planning for dropping the Python2 support in OpenStack
by Ghanshyam Mann 21 Nov '19

21 Nov '19

Hello Everyone, Python 2.7 is going to retire in Jan 2020 [1] and we planned to drop the python 2 support from OpenStack during the start of the Ussuri cycle[2]. Time has come now to start the planning on dropping the Python2. It needs to be coordinated among various Projects, libraries, vendors driver, third party CI and testing frameworks. * Preparation for the Plan & Schedule: Etherpad: https://etherpad.openstack.org/p/drop-python2-support We discussed it in TC to come up with the plan, execute it smoothly and avoid breaking any dependent projects. I have prepared an etherpad[3](mentioned above also) to capture all the points related to this topic and most importantly the draft schedule about who can drop the support and when. The schedule is in the draft state and not final yet. The most important points are if you are dropping the support then all your consumers (OpenStack Projects, Vendors drivers etc) are ready for that. For example, oslo, os-bricks, client lib, testing framework projects will keep the python2 support until we make sure all the consumers of those projects do not require py2 support. If anyone require then how long they can support py2. These libraries, testing frameworks will be the last one to drop py2. We have planned to have a dedicated discussion in TC office hours on the 24th Thursday #openstack-tc channel. We will discuss what all need to be done and the schedules. You do not have to drop it immediately and keep eyes on this ML thread till we get the consensus on the community-level plan and schedule. Meanwhile, you can always start pre-planning for your projects, for example, stephenfin has started for Nova[4] to migrate the third party CI etc. Cinder has coordinated with all vendor drivers & their CI to migrate from py2 to py3. * Projects want to keep the py2 support? There is no mandate that projects have to drop the py2 support right now. If you want to keep the support then key things to discuss are what all you need and does all your dependent projects/libs provide the support of py2. This is something needs to be discussed case by case. If any project wants to keep the support, add that in the etherpad with a brief reason which will be helpful to discuss the need and feasibility. Feel free to provide feedback or add the missing point on the etherpad. Do not forget to attend the 24th Oct 2019, TC office hour on Thursday at 1500 UTC in #openstack-tc. [1] https://pythonclock.org/ [2] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecatio… [3] https://etherpad.openstack.org/p/drop-python2-support [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010109.… -gmann

7 18

[manila] No IRC meeting on Nov 7 and Nov 21 2019
by Goutham Pacha Ravi 18 Nov '19

18 Nov '19

Hello Zorillas and interested stackers, Due to a part of our community attending the Open Infrastructure Summit+PTG (Nov 4-8, 2019) and KubeCon+CloudNativeCon (Nov 18-21, 2019), I propose that we cancel the weekly IRC meetings on Nov 7th and Nov 21st. If you'd like to discuss anything during these weeks, please chime in on freenode/#openstack-manila, or post to this mailing list. Thanks, Goutham

1 1

[ptg] Auto-generated etherpad links !
by Thierry Carrez 13 Nov '19

13 Nov '19

Hi everyone, The PTGbot grew a new feature over the summer. It now dynamically generates the list of PTG track etherpads. You can find that list at: http://ptg.openstack.org/etherpads.html If you haven't created your etherpad already, just follow the link there to create your etherpad. If you have created your track etherpad already under a different name, you can overload the automatically-generated name using the PTGbot. Just join the #openstack-ptg channel and (as a Freenode authenticated user) send the following command: #TRACKNAME etherpad <URL for your etherpad> Example: #keystone etherpad https://etherpad.openstack.org/p/awesome-keystone-pad That will update the link on that page automatically. Hoping to see you in Shanghai! -- Thierry Carrez (ttx)

2 1

[kuryr] [tc] kuryr project mission
by Michał Dulko 13 Nov '19

13 Nov '19

Hi, It's been more than a year after fuxi [1] and fuxi-kubernetes [2] were declared as retired. First one was supposed to integrate Docker with Cinder and Manila. The second was aiming to do the same with Kubernetes. I'm not really sure how to serve the first use case, but it seems like there are some alternatives [3], [4]. fuxi-kubernetes use case is now served by Cloud Provider OpenStack in Cinder [5] and Manila [6] CSI plugins, which seem like a much better fit. Currently the only maintained deliverables in kuryr project are kuryr, kuryr-libnetwork and kuryr-kubernetes. All of them are related to networking. Given that I'd like to propose rephrasing Kuryr mission statement from: > Bridge between container framework networking and storage models > to OpenStack networking and storage abstractions. to > Bridge between container framework networking models > to OpenStack networking abstractions. effectively getting storage out of project scope. Are there any thoughts or objections? Maybe someone sees a better phrasing? Thanks, Michał [1] https://opendev.org/openstack/fuxi [2] https://opendev.org/openstack/fuxi-kubernetes [3] https://rexray.readthedocs.io/en/stable/user-guide/schedulers/docker/plug-i… [4] https://github.com/j-griffith/cinder-docker-driver [5] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/usi… [6] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/usi…

5 4

[tc][horizon][all] Horizon plugins maintenance
by Ivan Kolodyazhny 13 Nov '19

13 Nov '19

Hi team, As you may know, we've got a pretty big list of Horizon Plugins [1]. Unfortunately, not all of them are in active development due to the lack of resources in projects teams. As a Horizon team, we understand all the reasons, and we're doing our best to help other teams to maintain plugins. That's why we're proposing our help to maintain horizon plugins. We raised this topic during the last Horizon weekly meeting [2] and we'll have some discussion during the PTG [3] too. There are a lot of Horizon changes which affect plugins and horizon team is ready to help: - new Django versions - dependencies updates - Horizon API changes - etc. To get faster fixes in, it would be good to have +2 permissions for the horizon-core team for each plugin. We helped Heat team during the last cycle adding horizon-core to the heat-dashboard-core team. Also, we've got +2 on other plugins via global project config [4] and via Gerrit configuration for (neutron-*aas-dashboard, tuskar-ui). Vitrage PTL agreed to do the same for vitrage-dashboard during the last meeting [5]. Of course, it's up to each project to maintain horizon plugins and it's responsibilities but I would like to raise this topic to the TC too. I really sure, that it will speed up some critical fixes for Horizon plugins and makes users and operators experience better. [1] https://docs.openstack.org/horizon/latest/install/plugin-registry.html [2] http://eavesdrop.openstack.org/meetings/horizon/2019/horizon.2019-10-16-15.… [3] https://etherpad.openstack.org/p/horizon-u-ptg [4] http://codesearch.openstack.org/?q=horizon-core&i=nope&files=&repos=opensta… [5] http://eavesdrop.openstack.org/meetings/vitrage/2019/vitrage.2019-10-23-08.… Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

9 9

[nova] spec is ready to review and for feature liasion needed
by Brin Zhang(张百林) 13 Nov '19

13 Nov '19

Hi everyone: This spec comment by melwitt, and already updated, wait for review and need a feature liaison. Proposal for a safer noVNC console with password authentication https://review.opendev.org/#/c/623120 Brin Zhang

2 2

[kolla][ptg] Kolla Ussuri PVPTG (Partially Virtual PTG)
by Radosław Piliszek 08 Nov '19

08 Nov '19

Hello Everyone, As you may already know, Kolla core team is mostly not present on summit in Shanghai. Instead we are organizing a PTG next week, 7-8th Nov (Thu-Fri), in Białystok, Poland. Please let me know this week if you are interested in coming in person. We invite operators, contributors and contributors-to-be to join us for the virtual PTG online. The time schedule will be advertised later. Please fill yourself in on the whiteboard [1]. New ideas are welcome. [1] https://etherpad.openstack.org/p/kolla-ussuri-ptg Kind regards, Radek aka yoctozepto

6 9

[nova][ptg] Virtual PTG
by Eric Fried 08 Nov '19

08 Nov '19

Hello nova contributors and other stakeholders. As you are aware, nova maintainers will be sparser than usual at the ussuri PTG. For that reason, and also because it promotes better inclusion anyway, I'd like us to do the majority of decision making via the mailing list. The PTG is still a useful place to talk through design ideas, but this will give those not attending a voice in the final direction. To that end, I call your attention to the etherpad [1]. As usual, list your topics there. And if your topic is something for which you only need (or wish to start with) in-person discussions (e.g. "I'd like to do $thing but could use some help figuring out $how"), you're done. But if what you're shooting for is discussion leading to some kind of decision, like... - My spec has been stalled because we can't decide among N different approaches; we need to reach a consensus. - My feature is really important; can we please prioritize it for ussuri? ...then in addition to putting your topic on the etherpad, please initiate a (separate) thread on this mailing list, including [nova][ptg] in your subject line. Some of these topics may be resolved before the PTG itself. Others may be discussed in Shanghai. However, even if a consensus is reached in person, expect that decision to be tentative pending closure of the ML thread. Thanks, efried [1] https://etherpad.openstack.org/p/nova-shanghai-ptg

3 3

CPU pinning blues
by Albert Braden 06 Nov '19

06 Nov '19

I'm following this document to setup CPU pinning on Rocky: https://www.redhat.com/en/blog/driving-fast-lane-cpu-pinning-and-numa-topol… I followed all of the steps except for modifying non-pinned flavors and I have one aggregate containing a single NUMA-capable host: root@us01odc-dev1-ctrl1:/var/log/nova# os aggregate list +----+-------+-------------------+ | ID | Name | Availability Zone | +----+-------+-------------------+ | 4 | perf3 | None | +----+-------+-------------------+ root@us01odc-dev1-ctrl1:/var/log/nova# os aggregate show 4 +-------------------+----------------------------+ | Field | Value | +-------------------+----------------------------+ | availability_zone | None | | created_at | 2019-10-30T23:05:41.000000 | | deleted | False | | deleted_at | None | | hosts | [u'us01odc-dev1-hv003'] | | id | 4 | | name | perf3 | | properties | pinned='true' | | updated_at | None | +-------------------+----------------------------+ I have a flavor with the NUMA properties: root@us01odc-dev1-ctrl1:/var/log/nova# os flavor show s1.perf3 +----------------------------+-------------------------------------------------------------------------+ | Field | Value | +----------------------------+-------------------------------------------------------------------------+ | OS-FLV-DISABLED:disabled | False | | OS-FLV-EXT-DATA:ephemeral | 0 | | access_project_ids | None | | disk | 35 | | id | be3d21c4-7e91-42a2-b832-47f42fdd3907 | | name | s1.perf3 | | os-flavor-access:is_public | True | | properties | aggregate_instance_extra_specs:pinned='true', hw:cpu_policy='dedicated' | | ram | 30720 | | rxtx_factor | 1.0 | | swap | 7168 | | vcpus | 4 | +----------------------------+-------------------------------------------------------------------------+ I create a VM with that flavor: openstack server create --flavor s1.perf3 --image NOT-QSC-CentOS6.10-19P1-v4 --network it-network alberttest4 but it goes to error status, and I see this in the logs: 2019-10-30 16:17:55.590 3248800 INFO nova.virt.hardware [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Computed NUMA topology CPU pinning: usable pCPUs: [[4], [5], [6], [7]], vCPUs mapping: [(0, 4), (1, 5), (2, 6), (3, 7)] 2019-10-30 16:17:55.595 3248800 INFO nova.virt.hardware [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Computed NUMA topology CPU pinning: usable pCPUs: [[0], [1], [2], [3], [4], [5], [6], [7]], vCPUs mapping: [(0, 0), (1, 1), (2, 2), (3, 3)] 2019-10-30 16:17:55.595 3248800 INFO nova.filters [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filter AggregateInstanceExtraSpecsFilter returned 0 hosts 2019-10-30 16:17:55.596 3248800 INFO nova.filters [req-d0c2de13-db23-41bd-8da3-34c68ff1d998 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filtering removed all hosts for the request with instance ID '73b1e584-0ce4-478c-a706-c5892609dc3f'. Filter results: ['RetryFilter: (start: 3, end: 3)', 'AvailabilityZoneFilter: (start: 3, end: 3)', 'CoreFilter: (start: 3, end: 2)', 'RamFilter: (start: 2, end: 2)', 'ComputeFilter: (start: 2, end: 2)', 'ComputeCapabilitiesFilter: (start: 2, end: 2)', 'ImagePropertiesFilter: (start: 2, end: 2)', 'ServerGroupAntiAffinityFilter: (start: 2, end: 2)', 'ServerGroupAffinityFilter: (start: 2, end: 2)', 'DifferentHostFilter: (start: 2, end: 2)', 'SameHostFilter: (start: 2, end: 2)', 'NUMATopologyFilter: (start: 2, end: 2)', 'AggregateInstanceExtraSpecsFilter: (start: 2, end: 0)'] It looks like my hypervisor is passing the hw:cpu_policy='dedicated' requirement but it is failing on "pinned=true" The interesting part of the problem is that if I add a second apparently identical hypervisor to the aggregate it starts working. I create s1.perf3 VMs and they land on us01odc-dev1-hv002 and the XML shows that they are correctly pinned. When us01odc-dev1-hv002 is full then they start failing again. What should I be looking for here? What could cause one apparently identical hypervisor to fail AggregateInstanceExtraSpecsFilter while another one passes? In the nova-compute log of the failing hypervisor I see this: 2019-10-31 10:43:01.147 1103 INFO nova.compute.resource_tracker [req-dda65a9c-9d0a-4888-b4cb-0bf4423dd2f3 - - - - -] Instance 4856d505-c220-4873-b881-836b5b75f7bb has allocations against this compute host but is not found in the database. 2019-10-31 10:43:01.148 1103 INFO nova.compute.resource_tracker [req-dda65a9c-9d0a-4888-b4cb-0bf4423dd2f3 - - - - -] Final resource view: name=us01odc-dev1-hv003.internal.synopsys.com phys_ram=128888MB used_ram=38912MB phys_disk=1208GB used_disk=297GB total_vcpus=8 used_vcpus=6 pci_stats=[] Openstack can't find a VM with UUID 4856d505-c220-4873-b881-836b5b75f7bb. There are no VMs on hv003 but I can create a non-pinned VM there and it works. Do I have a "phantom" VM that is consuming resources on hv003? How can I fix that?

2 5

[Neutron] cancel the L3 meeting today
by LIU Yulong 06 Nov '19

06 Nov '19

Hi all, Due to the Summit and PTG next week, I will cancel the Neutron L3 meeting today. So see you there, and travel safe. Thanks. LIU Yulong

1 1