November 2019 - openstack-discuss

RE: [nova][neutron][cyborg] Bandwidth (and accel) providers are broken if CONF.host is set
by Balázs Gibizer 22 Nov '19

22 Nov '19

On Fri, Nov 22, 2019 at 02:11, "Nadathur, Sundar" <sundar.nadathur(a)intel.com> wrote: >> -----Original Message----- >> From: Eric Fried <openstack(a)fried.cc> >> Sent: Thursday, November 21, 2019 3:28 PM > >> Action Summary >> ============== >> If the above sounds reasonable, it would entail the following >> actions: >> - Neutron(/Cyborg?): backportable patch to >> s/CONF.host/socket.gethostname()/ >> - Nova: GET /os-hypervisors*?service_host=X in a new microversion. >> - Neutron/Cyborg: master-only patch to do the logic described in >> `Upgrade >> Concerns`_ (though for now without the `elif` branch). > > Cyborg does use CONF.host today. We can use socket.gethostname() > instead. > > On a related node, the patch for ARQ binding [1] uses instance.host, > which comes from CONF.host. I'll change it to instance.node, which > comes from [2], which in turn comes from get_hostname() [3]. I'm not sure that for binding you need to use the instance.node. See my neutron port binding discussion in my response to Eric's original mail. > > [1] > https://review.opendev.org/#/c/631244/46/nova/compute/manager.py@2634 > [2] > https://github.com/openstack/nova/blob/master/nova/compute/manager.py#L9137 > [3] > https://github.com/openstack/nova/blob/1cd5563f2dd2b218db2422397c8aab394d48… > >> Thanks, >> efried > > > Thanks & Regards, > Sundar >

1 0

[nova][ops] heads up: change in behavior for resize and migrate server action APIs
by melanie witt 22 Nov '19

22 Nov '19

Hey all, Just wanted to give everyone a heads up about a recent change [1] in behavior for the resize [2] and migrate [3] server action APIs: https://docs.openstack.org/releasenotes/nova/unreleased.html#other-notes The scheduling step for resize and migrate has been changed from synchronous to asynchronous, so users will no longer receive a 400 error for NoValidHost. To discover whether the resize or migrate has failed during scheduling, users must use the server actions API [4], which is called "server events" in the openstackclient [5]. There is a WIP proposal [6] to add exception messages to events in the server actions API: https://review.opendev.org/694428 in order to communicate additional failure info to non-admin users. I think we'll want this as the server actions API becomes more of a source of information for the end user. Feel free reply here or visit the review to add your thoughts. Cheers, -melanie [1] https://review.opendev.org/693937 [2] https://docs.openstack.org/api-ref/compute/#resize-server-resize-action [3] https://docs.openstack.org/api-ref/compute/#migrate-server-migrate-action [4] https://docs.openstack.org/api-ref/compute/#servers-actions-servers-os-inst… [5] https://docs.openstack.org/python-openstackclient/latest/cli/command-object… [6] background: http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010775…

1 0

Re: All VMs fail when --max exceeds available resources
by Albert Braden 22 Nov '19

22 Nov '19

The other symptom is that the scheduler will send single VMs to a full hypervisor and overload it even though we have cpu_allocation_ratio and ram_allocation_ratio set to 1: root@us01odc-dev1-ctrl1:~# os hypervisor list --long +----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+ | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State | vCPUs Used | vCPUs | Memory MB Used | Memory MB | +----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+ | 1 | us01odc-dev1-hv003.internal.synopsys.com | QEMU | 10.195.116.16 | up | 42 | 16 | 161792 | 128888 | | 3 | us01odc-dev1-hv002.internal.synopsys.com | QEMU | 10.195.116.15 | up | 43 | 16 | 165888 | 128888 | | 4 | us01odc-dev1-hv001.internal.synopsys.com | QEMU | 10.195.116.14 | up | 38 | 16 | 161792 | 128888 | +----+------------------------------------------+-----------------+---------------+-------+------------+-------+----------------+-----------+ In the logs I see the scheduler returning 1 host: /var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.930 895454 DEBUG nova.filters [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filter NUMATopologyFilter returned 1 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:104 It weighs the host and reports negative RAM: /var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.930 895454 DEBUG nova.scheduler.filter_scheduler [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Weighed [WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 683008MB io_ops: 1 instances: 5, weight: 0.0]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454 Then it selects that host: /var/log/nova/nova-scheduler.log:2019-11-19 16:38:20.931 895454 DEBUG nova.scheduler.utils [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Attempting to claim resources in the placement API for instance 834dd112-26f8-424c-a92b-23423baa185a claim_resources /usr/lib/python2.7/dist-packages/nova/scheduler/utils.py:935 /var/log/nova/nova-scheduler.log:2019-11-19 16:38:21.567 895454 DEBUG nova.scheduler.filter_scheduler [req-0703f1f8-a52a-4fb6-a402-226dd25e9988 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: 834dd112-26f8-424c-a92b-23423baa185a] Selected host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 683008MB io_ops: 1 instances: 5 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346 The VM builds successfully and goes to ACTIVE. What should I be looking for here? Obviously I broke the scheduler, but my nova config is the same as the working cluster.

4 11

[nova][gate] Thoughts on working around bug 1853453?
by Matt Riedemann 21 Nov '19

21 Nov '19

I've been noticing these shelve/unshelve guest ssh fail due to dhcp lease issues quite a bit recently and wrote a bug and e-r query for it this morning: http://status.openstack.org/elastic-recheck/#1853453 The problem seems to stem from when these shelve tests run on multinode jobs and we shelve on one host and unshelve on another. I have a patch up to nova to force config drive in the nova-next job where this hits the most: https://review.opendev.org/#/c/695431 But that's just kind of a stab in the dark to take the metadata API out of the picture for cloud-init. If that doesn't help, and we don't know what is causing this or have ideas to debug it, we might need to consider making a change to shelve/unshelve testing in tempest such that we try to unshelve on original host. Now I realize that is unfortunate since the whole point of shelve offloading and unshelving is that you can land on another host and things are good, but if these tests continue to be a high failure rate in multinode jobs we probably need to consider workarounds if no one is going to dig into the failures and figure out what is going wrong. Thoughts? -- Thanks, Matt

2 1

[monasca] New team meeting time poll
by Witek Bedyk 21 Nov '19

21 Nov '19

Hello everyone, We would like to find the new time slot for the Monasca Team Meeting which suites you best. Please fill in the times which work for you in that poll [1] until next Wednesday. Thanks Witek [1] https://doodle.com/poll/ey6brvmbsubkxpp9

1 1

Re: [nova][ptg] Allow compute nodes to use DISK_GB from shared storage RP by using aggregate relationship
by Balázs Gibizer 21 Nov '19

21 Nov '19

On Thu, Nov 14, 2019 at 02:58, "Patil, Tushar" <Tushar.Patil(a)nttdata.com> wrote: > On 11/13/2019 8:34 AM, Sylvain Bauza wrote: >>> Me too. To be clear, I don't think operators would modify the >>> above but >>> if so, they would need reshapes. > >> Maybe not, but this is the kind of detail that should be in the >> spec and >> functional tests to make sure it's solid since this is a big >> architectural change in nova. > > It depends on how the aggregates are created on the nova and > placement side. > > A) From placement point of view, operator can create a new aggregate > and add shared storage RP to it (tag MISC_SHARES_VIA_AGGREGATE trait > to this RP). The newly created valid UUID would then be set in the > config option ``sharing_disk_aggregate`` on the compute node side. > This aggregate UUID wouldn't be present in the nova aggregate. so > it's not possible to add host to the nova aggregate unless a new > aggregate is created on nova side. > > B) If nova aggregates are synced to the placement service and say > below is the picture: > > Nova: > > Agg1 - metadata (pinned=True) > - host1 > - host2 > > Now, operator adds a new shared storage RP to Agg1 on placement side > and then set Agg1 UUID in ``sharing_disk_aggregate`` on compute nodes > along with ``using_shared_disk_provider`=True``, then it would add > compute node RP to the Agg1 on the placement without any issues but > when you want to reverse the configuration, > using_shared_disk_provider=False, then it not that straight to remove > the host from the placement/nova aggregate because there would be > other traits set to compute RPs which could cause those functions > stop working. For me from the sharing disk provider feature perspective the placement aggregate that is needed for the sharing to work, and any kind of nova host aggregate (either synced to placement or not) is independent. The placement aggregate is a must for the feature. On top of that if the operator wants to create a nova host aggregate as well and sync it to placement then at the end there will be two, independent placement aggregates. One to express the sharing relationship and one to express a host aggregate from nova. These two aggregate will not be the same as the first one will have the sharing provider in it while the second one doesn't. gibi > > We had same kind of discussion [1] when implementing forbidden > aggregates where we want to sync traits set to the aggregates but > later it was concluded that operator will do it manually. > > I will include the details Matt has pointed out in this email in my > next patchset. > > [1] : > http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006950.html > > Regards, > tpatil > > > > ________________________________________ > From: Matt Riedemann <mriedemos(a)gmail.com> > Sent: Wednesday, November 13, 2019 11:41 PM > To: openstack-discuss(a)lists.openstack.org > Subject: Re: [nova][ptg] Allow compute nodes to use DISK_GB from > shared storage RP by using aggregate relationship > > On 11/13/2019 8:34 AM, Sylvain Bauza wrote: >> Me too. To be clear, I don't think operators would modify the above >> but >> if so, they would need reshapes. > > Maybe not, but this is the kind of detail that should be in the spec > and > functional tests to make sure it's solid since this is a big > architectural change in nova. > > -- > > Thanks, > > Matt > > Disclaimer: This email and any attachments are sent in strictest > confidence for the sole use of the addressee and may contain legally > privileged, confidential, and proprietary data. If you are not the > intended recipient, please advise the sender by replying promptly to > this email and then delete and destroy this email and any attachments > without any further use, copying or forwarding. >

3 2

[cinder] meeting time change poll results
by Brian Rosmaita 21 Nov '19

21 Nov '19

Here are the results of the meeting time change poll: 16 responses - current time (1600 UTC): 19% unlikely to attend, 25% love it - one hour earlier (1500 UTC): 13% unlikely to attend, 19% love it - two hours earlier (1400 UTC): 0% unlikely to attend, 56% love it Thus, we will reschedule the Cinder weekly meeting to occur each Wednesday at 1400 UTC beginning with the meeting on 4 December 2019. Our current meeting channel is occupied at that time. I've submitted a patch to hold the meeting in #openstack-meeting-4: https://review.opendev.org/695339 I'll send out another email when that patch has been approved and the new meeting location has been verified. cheers, brian

1 0

[all][tc] Planning for dropping the Python2 support in OpenStack
by Ghanshyam Mann 21 Nov '19

21 Nov '19

Hello Everyone, Python 2.7 is going to retire in Jan 2020 [1] and we planned to drop the python 2 support from OpenStack during the start of the Ussuri cycle[2]. Time has come now to start the planning on dropping the Python2. It needs to be coordinated among various Projects, libraries, vendors driver, third party CI and testing frameworks. * Preparation for the Plan & Schedule: Etherpad: https://etherpad.openstack.org/p/drop-python2-support We discussed it in TC to come up with the plan, execute it smoothly and avoid breaking any dependent projects. I have prepared an etherpad[3](mentioned above also) to capture all the points related to this topic and most importantly the draft schedule about who can drop the support and when. The schedule is in the draft state and not final yet. The most important points are if you are dropping the support then all your consumers (OpenStack Projects, Vendors drivers etc) are ready for that. For example, oslo, os-bricks, client lib, testing framework projects will keep the python2 support until we make sure all the consumers of those projects do not require py2 support. If anyone require then how long they can support py2. These libraries, testing frameworks will be the last one to drop py2. We have planned to have a dedicated discussion in TC office hours on the 24th Thursday #openstack-tc channel. We will discuss what all need to be done and the schedules. You do not have to drop it immediately and keep eyes on this ML thread till we get the consensus on the community-level plan and schedule. Meanwhile, you can always start pre-planning for your projects, for example, stephenfin has started for Nova[4] to migrate the third party CI etc. Cinder has coordinated with all vendor drivers & their CI to migrate from py2 to py3. * Projects want to keep the py2 support? There is no mandate that projects have to drop the py2 support right now. If you want to keep the support then key things to discuss are what all you need and does all your dependent projects/libs provide the support of py2. This is something needs to be discussed case by case. If any project wants to keep the support, add that in the etherpad with a brief reason which will be helpful to discuss the need and feasibility. Feel free to provide feedback or add the missing point on the etherpad. Do not forget to attend the 24th Oct 2019, TC office hour on Thursday at 1500 UTC in #openstack-tc. [1] https://pythonclock.org/ [2] https://governance.openstack.org/tc/resolutions/20180529-python2-deprecatio… [3] https://etherpad.openstack.org/p/drop-python2-support [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010109.… -gmann

7 18

[cinder] Ussuri Virtual PTG next week
by Brian Rosmaita 21 Nov '19

21 Nov '19

As announced at today's Cinder weekly meeting, the Virtual PTG will be held next week on two days: - Monday 25 November 1500-1700 UTC - Wednesday 27 November 1500-1700 UTC The planning etherpad is: https://etherpad.openstack.org/p/cinder-ussuri-virtual-ptg-planning Feel free to add topics. They can be new topics or something you want to follow up on from the Shanghai PTG. Format will be the usual, that is, we'll start at the beginning, give each topic what it takes, and finish when we are done. We'll use BlueJeans video conferencing: https://bluejeans.com/3228528973 If you haven't used BlueJeans before, ping me in #openstack-cinder and I can do a quick meeting with you to make sure you can connect OK. cheers, brian

1 0

All VMs fail when --max exceeds available resources
by Albert Braden 21 Nov '19

21 Nov '19

I was experimenting in our dev cluster with CPU pinning and filters. After I was done I ran Ansible to put everything back like it was, but the scheduler is broken in 2 ways and I can't find the problem in my config. The first symptom is that if I use -max to create more VMs than the hypervisors can support, all of them go to ERROR. Before I changed things, --max would fill the hypervisors and only the extra VMs would go to ERROR. I'll email separately about the other one; this is already getting long. When I look at the logs, I see the "Starting to schedule" line with the list of instance UUIDs, and then "Attempting to claim resources" log entries. Hosts are selected; they are starting with 0 instances: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:56.880 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: ef1f7493-f792-4c3b-bf50-8e68b3d553ac] Selected host: (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 120696MB disk: 986112MB io_ops: 0 instances: 0 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346 Scheduler starting with the correct number of HV: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:56.888 1409571 DEBUG nova.filters [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Starting with 3 host(s) get_filtered_objects /usr/lib/python2.7/dist-packages/nova/filters.py:70 I see the HV being weighed and the number of instances on each HV increasing from 0: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:57.106 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Filtered [(us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: 85880MB disk: 880640MB io_ops: 1 instances: 2, (us01odc-dev1-hv001, us01odc-dev1-hv001.internal.synopsys.com) ram: 89976MB disk: 932864MB io_ops: 0 instances: 1, (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 89976MB disk: 929792MB io_ops: 1 instances: 1] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:435 us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:57.107 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Weighed [WeighedHost [host: (us01odc-dev1-hv001, us01odc-dev1-hv001.internal.synopsys.com) ram: 89976MB disk: 932864MB io_ops: 0 instances: 1, weight: 2.40576402895], WeighedHost [host: (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 89976MB disk: 929792MB io_ops: 1 instances: 1, weight: 1.11693447844], WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: 85880MB disk: 880640MB io_ops: 1 instances: 2, weight: 0.961725168625]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454 The number of instances goes beyond the HV capacity: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:57.602 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Weighed [WeighedHost [host: (us01odc-dev1-hv001, us01odc-dev1-hv001.internal.synopsys.com) ram: 59256MB disk: 876544MB io_ops: 1 instances: 2, weight: 1.17395901159], WeighedHost [host: (us01odc-dev1-hv003, us01odc-dev1-hv003.internal.synopsys.com) ram: 59256MB disk: 873472MB io_ops: 2 instances: 2, weight: 0.435549629144], WeighedHost [host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: 55160MB disk: 824320MB io_ops: 2 instances: 3, weight: 0.292945361349]] _get_sorted_hosts /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:454 Hosts are still being selected, but they are way over capacity already: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:58.696 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: 579dd4e2-d5d5-445f-905f-84cfd93146f6] Selected host: (us01odc-dev1-hv002, us01odc-dev1-hv002.internal.synopsys.com) ram: -6280MB disk: 711680MB io_ops: 4 instances: 5 _consume_selected_host /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:346 Then we start to see the warnings: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:58.817 1409571 WARNING nova.scheduler.client.report [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Unable to submit allocation for instance 5de08185-1ef2-4c92-8a19-5f09ec27be71 (409 {"errors": [{"status": 409, "request_id": "req-6cdd0b7a-bdbd-486f-9792-20840ea4a72e", "code": "placement.undefined_code", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'MEMORY_MB' on resource provider 'f20fa03d-18f4-486b-9b40-ceaaf52dabf8'. The requested amount would exceed the capacity. ", "title": "Conflict"}]}) And then they all fail and are deleted: us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.057 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Unable to successfully claim against any host. _schedule /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:242 us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.058 1409571 DEBUG nova.scheduler.filter_scheduler [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Cleaning up allocations for [u'ef1f7493-f792-4c3b-bf50-8e68b3d553ac', u'80afcfd7-ce25-4fc4-8b0d-b581a31b87bd', u'22cf9509-39e8-456a-b32c-e950cc597266', u'd6fc0b66-cd44-410d-96c9-590c22f1e21b', u'713a12fb-8bbf-4eac-ae02-68cb007fa34e', u'30c70c4d-1447-4481-bdc5-816835180ac6', u'be955f43-ac56-4531-ada8-16ce966211c7', u'b91820f6-775e-4abb-b28e-5b9b065819c2', u'83472954-2197-4e65-b7e7-1fe892f28458', u'd862bd74-494b-4c29-94ad-b80fc6c113e8', u'80a69dba-8011-403f-87fb-c39ef17ba467', u'c440d8d1-4da2-4c27-af9f-dd5afa19d083', u'5b301d69-a4f1-4393-8d67-4286c9873490', u'579dd4e2-d5d5-445f-905f-84cfd93146f6'] _cleanup_allocations /usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py:299 us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.124 1409571 INFO nova.scheduler.client.report [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Deleted allocation for instance ef1f7493-f792-4c3b-bf50-8e68b3d553ac us01odc-dev1-ctrl3:/var/log/nova/nova-scheduler.log:2019-11-19 11:39:59.190 1409571 INFO nova.scheduler.client.report [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Deleted allocation for instance 80afcfd7-ce25-4fc4-8b0d-b581a31b87bd And the errors: us01odc-dev1-ctrl1:/var/log/nova/nova-conductor.log:2019-11-19 11:40:00.201 1801903 WARNING nova.scheduler.utils [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] Failed to compute_task_build_instances: No valid host was found. There are not enough hosts available. us01odc-dev1-ctrl1:/var/log/nova/nova-conductor.log:2019-11-19 11:40:00.205 1801903 WARNING nova.scheduler.utils [req-892201d0-6b04-4e92-a347-00285725fbed 2cb6757679d54a69803a5b6e317b3a93 474ae347d8ad426f8118e55eee47dcfd - default 7d3a4deab35b434bba403100a6729c81] [instance: 5b301d69-a4f1-4393-8d67-4286c9873490] Setting instance to ERROR state.: NoValidHost_Remote: No valid host was found. There are not enough hosts available.

1 0

[tc] Recent updates and action items
by Jean-Philippe Evrard 20 Nov '19

20 Nov '19

Hello TC members, There are still business opportunities merging for 2019. The process for 2020 need to start soon. We have action items from the Summit and PTG [1] to take care. Please review etherpads and check if any part actions that you already been assigned or whould like to help. Here are a few action items already if you want to help: - ttx: will be working on the technical vision reflection, as we arrive at the end of the road (2019). - gmann: will be working on the multiple goals he has, including python versions. A little help would be welcomed I guess :) - mnaser: will be working on telemetry and an oslo project to expose metrics for prometheus. (If I didn't get that one wrong) - jungleboyj: will write a blog post about the analysis of the Foundation user survey. We should read it when it's ready and prepare the next one - mugsie: is writing a new proposal for release naming based on what we discussed at the summit - We all should think about stable policy and what to do about it, based on the feedback from TheJulia/smcginnis/release+requirements team. Maybe diablo_rojo can follow this? - everyone: we should promote the first contact sig to organisations that might need help on getting things achieved in OpenStack. - zaneb: will propose documentation of the process on how TC members should vote on conflicting resolutions when there is no predetermined consensus - mnaser: will be the point of contact for the infra matters of static hosting - ttx: will investigate a merge tc/uc - evrardjp: will start the new ideas concept that was floated before the summit and got traction during the summit. PowerVMStackers is now a SIG [2]. However, mnaser suggested to move to a multi-arch SIG. Mohammed, do you want to get started on this? V release naming process is now started [3]. [1] https://etherpad.openstack.org/p/PVG-TC-PTG [2] https://review.opendev.org/#/c/680438/ [3] https://review.opendev.org/#/c/693266/ Regards, Jean-Philippe & Rico

3 2

[release][stable][telemetry]Please add Rong Zhu to ceilometer-stable-maint group
by Rong Zhu 20 Nov '19

20 Nov '19

Hi Stable Maintenance Core team, I am the current Telemetry PTL, could you please add me to ceilometer-stable-maint group. And please also add Lingxian Kong to this group. -- Thanks, Rong Zhu

3 2

[glance] glance_store tests failed
by Naohiro Sameshima（鮫島直洋）(Group) 20 Nov '19

20 Nov '19

Hi, When I run a test with `tox -e py37` in glance_store, two tests failed. The command what I ran is below. 1. git clone https://opendev.org/openstack/glance_store.git 2. tox -e py37 Is there something wrong with how to run test? Thanks & Best Regards, ============================== Failed 2 tests - output below: ============================== glance_store.tests.unit.test_filesystem_store.TestStore.test_add_check_metadata_list_with_valid_mountpoint_locations -------------------------------------------------------------------------------------------------------------------- Captured traceback: ~~~~~~~~~~~~~~~~~~~ b'Traceback (most recent call last):' b' File "/Users/sameshima/glance_store/glance_store/tests/unit/test_filesystem_store.py", line 215, in test_add_check_metadata_list_with_valid_mountpoint_locations' b' self.assertEqual(in_metadata[0], metadata)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual' b' self.assertThat(observed, matcher, message)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat' b' raise mismatch_error' b"testtools.matchers._impl.MismatchError: {'id': 'abcdefg', 'mountpoint': '/tmp'} != {}" b'' glance_store.tests.unit.test_multistore_filesystem.TestMultiStore.test_add_check_metadata_list_with_valid_mountpoint_locations ------------------------------------------------------------------------------------------------------------------------------ Captured traceback: ~~~~~~~~~~~~~~~~~~~ b'Traceback (most recent call last):' b' File "/Users/sameshima/glance_store/glance_store/tests/unit/test_multistore_filesystem.py", line 276, in test_add_check_metadata_list_with_valid_mountpoint_locations' b' self.assertEqual(in_metadata[0], metadata)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 411, in assertEqual' b' self.assertThat(observed, matcher, message)' b' File "/Users/sameshima/glance_store/.tox/py37/lib/python3.7/site-packages/testtools/testcase.py", line 498, in assertThat' b' raise mismatch_error' b"testtools.matchers._impl.MismatchError: {'id': 'abcdefg', 'mountpoint': '/tmp'} != {'store': 'file1'}" This email and all contents are subject to the following disclaimer: https://hello.global.ntt/en-us/email-disclaimer

3 3

[kolla][gnocchi] Issue on Gnocchi metricd when using Ceph as backend but with cache tier.
by Eddie Yen 20 Nov '19

20 Nov '19

Hi everyone, I'm not sure if it's good to ask at this place but still want to make sure that some one has a same issue as me. Currently we're using Kolla Openstack (R version), and Gnocchi version is 4.3.2 (ubuntu-source). The issue is, we always get below error trace in gnocchi-metricd after launched VM or uploaded images that let Ceilometer to create metrics in few environments. https://pastebin.com/fPZSFWaE After few investigations, we found that this only happens on the environment which enabled cache tier in Ceph. And also found the gnocchi cache pool only got 14 Bytes data inside, and no data inside the OSD pool one. We tried remove the cache tier from gnocchi pool then restart the gnocchi services. Then everything become normal. Can saw Gnocchi work normally to write data into Ceph pool. Does anyone had a same problem? Eddie.

1 0

[Watcher] irc meeting at 8:00 UTC today
by Canwei Li 20 Nov '19

20 Nov '19

Hi,Watcher team meeting is in the #openstack-meeting-alt channel. The agenda is available on https://wiki.openstack.irg/wiki/Watcher_Meeting_Agenda Thanks! Canwei Li 发自139邮箱

2 1

[all][tc] What happened in OpenStack Governance last month
by Jean-Philippe Evrard 20 Nov '19

20 Nov '19

Hello everyone, I hope you're all back from the summit to a normal life (nod to all the people at conferences, like KubeCon or devops days)! Here are a few things you might want to know about... First, for Python2.7 and Python3: We now removed references to Python 2.7 in the PTI (Project Testing Interface) [1]. In Ussuri we no longer require projects to test for python 2.7, so remove it from the list of requirements in the PTI. You can find current PTI for python in [2]. For py2 to py3 migration timing, there is now a ML [4] and etherpad[6] to discuss about specific schedule (now out in [6]), please join our discussions! We need each teams to pay attention on their python dependency and check if that schedule works. The release schedule will be update accordingly. We recently merged updates to the drop py2 goal too [9]. Then, for changes in projects, SIGs or adjacent communities: - If you didn't know: Airship is now confirmed as official project under OSF [3]. Congrats Airship team! - In OpenStack projects, we now have new barbican-ui, and manila- ganesha charms repositories. Microversion parse was adopted by the oslo team [7], and the API site was un-retired for redirect reasons. - Large scale SIG is under plan [8]. Please join, if you're interested on it. - PowerVM is now a SIG instead of a project. In the future, we might want to have a Multi-Arch SIG. - The Technical Writing SIG is born, replacing the openstack docs project. - Designate was promoted as a 2019 "business opportunity" for investment. We are now thinking for 2020. Thank you all for reading until this line! ;) [1] https://review.opendev.org/#/c/686582 [2] https://governance.openstack.org/tc/reference/pti/python.html [3] http://lists.openstack.org/pipermail/foundation/2019-October/002804.html [4] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010142.… [5] http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2019… [6] https://etherpad.openstack.org/p/drop-python2-support [7]: https://review.opendev.org/#/c/689754/ [8] http://lists.openstack.org/pipermail/openstack-discuss/2019-November/010757… [9]: https://review.opendev.org/#/c/694161/ Regards, Jean-Philippe & Rico

1 0

On next minimum libvirt / QEMU versions for "V" release
by Kashyap Chamarthy 20 Nov '19

20 Nov '19

Heya, The last time we incremented versions for libvirt and QEMU was during the "Stein" release[1]. For "Train" we didn't do any. Although we advertized NEXT_MIN_{LIBVIRT,QEMU} versions for "Train" release to be libvirt 4.0.0 and QEMU 2.11.0, but we actually didn't bump; we'll do that for "Ussuri". But before we bump the versions for "Ussuri", we need to pick NEXT_MIN versions for the "V" release. Based on the updated the DistroSupportMatrix page[2], it looks we can pick the next libvirt and QEMU versions for "V" release to the following: libvirt: 5.0.0 [GAed on: 15-Jan-2019] QEMU: 4.0.0 [GAed on: 24-Apr-2019] I have the initial patch here[3] for comments. Debian, Fedora, Ubuntu[4], CentOS, RHEL currently already ship the above versions (actually, higher than those). And it is reasonable to assume -- but let's confirm below -- that openSUSE, SLES, and Oracle Linux would also have the above versions available by "V" release time. Action Items for Linux Distros ------------------------------ (a) Oracle Linux: Please update your libvirt/QEMU versions for Oracle Linux 8? I couldn't find anything related to libvirt/QEMU here: https://yum.oracle.com/oracle-linux-8.html. (My educated guess is: the versions roughly match what's in CentOS/RHEL.) (b) openSUSE and SLES: Same request as above. Andreas Jaegaer said on #openstack-infra that the proposed versions for 'V' release should be fine for SLES. (And by extension open SUSE, I assume.) - - - Assuming Oracle Linux and SLES confirm, please let us know if there are any objections if we pick NEXT_MIN_* versions for the OpenStack "V" release to be libvirt: 5.0.0 and QEMU: 4.0.0. Comments / alternative proposals welcome :-) [1] https://opendev.org/openstack/nova/commit/489b5f762e -- Pick next minimum libvirt / QEMU versions for "T" release, 2018-09-25) [2] https://wiki.openstack.org/wiki/LibvirtDistroSupportMatrix [3] https://review.opendev.org/694821 -- [RFC] Pick NEXT_MIN libvirt/QEMU versions for "V" release [4] For Ubuntu, I updated the versions based on what is in the Cloud Archive repo for "Bionic" (the LTS) release: http://reqorts.qa.ubuntu.com/reports/ubuntu-server/cloud-archive/train_vers… -- /kashyap

2 3

[scientific-sig] Meeting CANCELLED - SC19
by Stig Telfer 20 Nov '19

20 Nov '19

Hi All - Unfortunately, we must cancel this week’s meeting as all the SIG chairs and many of the members are currently in Denver for Supercomputing 2019. Apologies, Stig

1 0

Stable branch tox now installed with Python 3 in gate Bionic nodes
by Ian Wienand 20 Nov '19

20 Nov '19

Hello, In the infra meeting we discussed that tox is installed with Python 3 on Bionic hosts in the gate now [1]. This can lead to tox jobs that do not specify a basepython running under python3. This was an unintentional change that came from [2] which uses the python version diskimage-builder thinks is the default on the platform to install tox. On Bionic this is Python 3. (The reasoning behind this was because of CentOS 8; where we are shipping a true Python3 only distribution) Opinion was mixed; this can be fixed by setting basepython=2 in the tox.ini. It's arguable this is the most correct solution and what should have been done, as it makes it agnostic to platform changes. This includes anyone who might be running *outside* the gate trying to replicate jobs. However, I'm sympathetic to the argument that things changing on stable jobs is very annoying too. We could put in a workaround to install tox under python2 on Bionic, restoring the old behaviour. Personally I'm of the opinion this just hides what is a real failure; that the test is only compatible with Python 2 but doesn't specify it. I feel like we should come down on the side of just fixing tests so they are isolated from this, be it in the gate or otherwise. Better to fix the gate quickly than have potential users struggling with running tests in incompatible environments, for mine. However, I'll take advice if the problems are just too big to fix piecemeal, and we need a workaround. Thanks, -i [1] http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-11-19-19.05.l… [2] https://review.opendev.org/#/c/686524/

1 0

[all][tc][forum] Summarizing Ussuri cycle community goal discussion in Forum and PTG
by Ghanshyam Mann 20 Nov '19

20 Nov '19

Hello Everyone, We discussed Ussuri goals in Forum as well as in PTG. I would like to summarize the discussions here. * Forum[1]: Three goals were discussed in details: 1. Drop Python 2.7 Support We discussed keeping the CI/CD support for Swift which is the only project keeping the py2 support. Swift needs the devstack to keep installing on py2 env with the rest of the services on py3 (same as old jobs when Swift was on py2 by default on devstack). There is no oslo dependency from swift and all the other dependency will be capped for py2 version. Requirements check job currently checks that if openstack/requirements list two entries for a requirement (one for <=2.7 and one for >) that the repo under test also has both entries. smcginnis already pushed the changes[2] to handle dual python version requirements. Everything else will go as discussed in ML[3] and this is already accepted as Ussuri goal[4]. 2. Project Specific New Contributor & PTL Docs - As per feedback in Forum sessions, this is a good goal which will make documentation more consistent. All the projects should edit their contributor.rst to follow a more complete template and adjust/add PTL documentation. - This is accepted as a pre-approved as Ussuri goal - Kim Hindhart is working on getting EU funding for people to work on OpenStack and they like consistent documentation. - diablo_rojo already updated the goal proposal patch and it is up to get wider feedback. 3. Switch remaining legacy jobs to Zuul v3 and drop legacy support - Grenade job is not yet on zuulv3 which is required to finish first. - Few projects waiting for big projects finishing the zuulv3 migration first. - This needs more work and can be a "pre-approved" thing for V, and this would be split to focus on the Grenade work in U. Other than above 3 goals, there were few more ideas for goal candidate and good to go in goal backlogs etherpad: - cdent: stop using paste, pastedeploy and WSME Note from Chris: This does not need to be a community goal as such but requires the common solution from TC WSME is still used, has contributions, and at least a core or two - cmurphy: Consistent and secure default policies https://etherpad.openstack.org/p/PVG-keystone-forum-policy Going with pop-up team first - support matrix documentation to be consistent across projects. going with pop-up team (fungi can propose the pop-up team in governance) first Richard Pioso (rpioso) to help fungi on this once consistent framework is identified, the pop-up team can expire with the approval of a related cycle goal for implementing it across remaining projects * PTG[5]: We all agreed to select the Ussuri goal asap because the Ussuri cycle already started and projects are waiting for the final goals. TC discussed this further in PTG and agreed on below two goals to be selected as the Ussuri goal. 1. Drop Python 2.7 Support - Already Accepted. 2. Project Specific New Contributor & PTL Docs - Under Review Goal "Switch remaining legacy jobs to Zuul v3 and drop legacy support" will be pre-selected for V cycle, that does not mean to stop the review on proposed goal or any ongoing work. All ongoing efforts will continue on this. [1] https://etherpad.openstack.org/p/PVG-ussuri-goal-forum [2] https://review.opendev.org/#/c/693631/ [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010371.… [4] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html [5] L76 https://etherpad.openstack.org/p/PVG-TC-PTG -gmann

1 0

[magnum] Kubernetes cluster issue
by Namrata Sitlani 19 Nov '19

19 Nov '19

Hello Folks, >From Thursday last week (Nov 14), Magnum is unable to spin up working Kubernetes clusters. We run on Rocky Openstack release. All our Kubernetes pods show CrashLoopBackOff status. We use the following commands to create the Kubernetes clusters : http://paste.openstack.org/show/786348/ . The deployment fails with the following output : http://paste.openstack.org/show/786287/. We tried deploying Kubernetes v1.13.12, v1.14.8, v1.15.5 and v1.16.2 without success. However, if we use version v1.14.6 we can successfully deploy our clusters. Unfortunately, we cannot use v1.14.6 in production because it is not patched for the CVE-2019-11253 vulnerability . Since this stopped working for us on Thursday, we think that the image update https://hub.docker.com/r/openstackmagnum/kubernetes-apiserver/tags done 6 days ago is the culprit. (We previously deployed clusters with versions v1.15.5 and v1.14.8 successfully.) Can you please confirm our findings and help us find a way forward? We can provide more logs if needed. Thank you very much, Namrata Sitlani

2 1

[RabbitMQ][cinder] Listen to messages
by Blom, Merlin, NMU-OI 19 Nov '19

19 Nov '19

Hey there, it seems to me as if ask.openstack.org is down, so I ask my question here: I'd like to listen to oslo messages from cinder as I do for neutron and octavia to know what is going on. For me the following code worked for neutron: EXCHANGE_NAME = os.getenv('EXCHANGE_NAME', 'neutron') ROUTING_KEY = os.getenv('ROUTING_KEY', 'notifications.info') QUEUE_NAME = os.getenv('QUEUE_NAME', 'messaging_queue') BROKER_URI = os.getenv('BROKER_URI', 'UNDEFINED') BROKER_PASSWORD = os.getenv('BROKER_PASSWORD', '') class Messages(ConsumerMixin): def __init__(self, connection): self.connection = connection return def get_consumers(self, consumer, channel): exchange = Exchange(EXCHANGE_NAME, type="topic", durable=False) queue = Queue(QUEUE_NAME, exchange, routing_key=ROUTING_KEY, durable=False, auto_delete=True, no_ack=True) return [consumer(queues=[queue], callbacks=[self.on_message])] def on_message(self, body, message): try: print(message) except Exception as e: log.info(repr(e)) if __name__ == "__main__": log.info("Connecting to broker {}".format(BROKER_URI)) with BrokerConnection(hostname=BROKER_URI, userid='messaging', password=BROKER_PASSWORD, virtual_host='/'+EXCHANGE_NAME, heartbeat=4, failover_strategy='round-robin') as connection: Messaging(connection).run() BrokerConnection.connection.close() But on the cinder vhost (/cinder) I can't find an exchange that the code is working on. (cinder, cinder-backup, .) I tried using the rabbitmq tracer: <https://www.rabbitmq.com/firehose.html> https://www.rabbitmq.com/firehose.html And got all the cinder messages but I don't want to use it in production because of performance issues. Does anyone have an idea how to find the correct exchange for the notification info queue in cinder? Cheers, Merlin

2 2

Trove Image issue
by rabia.shaheen＠xflowresearch.com 19 Nov '19

19 Nov '19

Hi Team, I have deployed kolla-ansible(Stein) with Trove enable and using Trove prebuild images to build the Database VM but VM is constantly stuck in build state. I am not sure how to use prebuild image key which is in https://opendev.org/openstack/trove/src/branch/master/integration/scripts/fi les/keys folder. Can you please guide me regarding the key usage for prebuild images (http://tarballs.openstack.org/trove/images/) To build my own trove image on kolla-ansible, is there any specific guide available for it? Warm Regards, Rabia Shaheen Lead Engineer, xFlow Research Inc. +923075462720 (GMT+5) <mailto:rabia.shaheen@xflowresearch.com> rabia.shaheen(a)xflowresearch.com <http://www.xflowresearch.com/> www.xflowresearch.com

2 1

how to update back-end storage online
by yj x 19 Nov '19

19 Nov '19

Hi, I want to implement a qemu block driver which like qeme/block/iscsi.c. Here are my questions: some targets have be attached to guest system, and then the targets info have changed. How can I update the connection to targets online? I mean the disk in guest system does not change, just update the connection to the back-end storage, and the operation can't affect guest system. I can't find a nova-api or libvirt-api to meet my needs now. Does anyone help me? Thanks.

1 0

[keystone] post-PTG virtual meeting reminder
by Colleen Murphy 19 Nov '19

19 Nov '19

Hi keystoners, As a reminder, we'll be holding our post-PTG meeting tomorrow at 14:00 UTC (with the daylight savings time change that makes it 6:00 PST (RIP me) / 9:00 EST / 19:30 IST). Last week I briefly floated the idea of rescheduling it but quickly decided there was not enough notice, so we're holding it at the day and time that we've had planned for the last few weeks. Our agenda and notes: https://etherpad.openstack.org/p/keystone-shanghai-ptg We'll try jitsi.org again, I think the technical issues we were having last time were on my end and not related to the platform: https://meet.jit.si/keystone-ptg Please please please review the roadmap board and assign yourself to items you have been working on and update their status: https://tree.taiga.io/project/keystone-ussuri-roadmap/kanban . We'll be going over the board together tomorrow, and updating it ahead of time will save us time as a group. Colleen

1 0

[manila] No IRC meeting on Nov 7 and Nov 21 2019
by Goutham Pacha Ravi 18 Nov '19

18 Nov '19

Hello Zorillas and interested stackers, Due to a part of our community attending the Open Infrastructure Summit+PTG (Nov 4-8, 2019) and KubeCon+CloudNativeCon (Nov 18-21, 2019), I propose that we cancel the weekly IRC meetings on Nov 7th and Nov 21st. If you'd like to discuss anything during these weeks, please chime in on freenode/#openstack-manila, or post to this mailing list. Thanks, Goutham

1 1

Shanghai PTG Team Photos
by Kendall Waters 18 Nov '19

18 Nov '19

Hi everyone, Thank you for attending the Project Teams Gathering in Shanghai! If your team took a team picture, you can find a copy of the photo file in this Dropbox folder: https://www.dropbox.com/sh/1my6wdtuc1hf58o/AACU49pjWxzFNzcZJgjLG8n1a?dl=0 If you are unable to open Dropbox, please send me an email with which team photo you are looking for and I can send you the file directly. Cheers, Kendall Kendall Waters OpenStack Marketing & Events kendall(a)openstack.org

1 0

[Openstack] FOSDEM 2020 Virtualization & IaaS Devroom CfP
by Piotr Kliczewski 18 Nov '19

18 Nov '19

Friendly reminder that there are 2 weeks before the submission deadline. Room day update: This year Virt and IaaS room will be on the 2nd of February. See you all at FOSDEM!

1 0

Scheduler sends VM to HV that lacks resources
by Albert Braden 18 Nov '19

18 Nov '19

If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU." https://paste.fedoraproject.org/paste/6N3wcDzlbNQgj6hRApHiDQ I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU." https://paste.fedoraproject.org/paste/lGlVpfbB9C19mMzrWQcHCQ I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard: enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full: https://paste.fedoraproject.org/paste/6SX9pQ4V1KnWfQkVnfoHOw This is the command line I used: openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB

5 14

[neutron][drivers] Drivers meeting cancel
by Slawek Kaplonski 18 Nov '19

18 Nov '19

Hi, I can't attend next drivers meeting on Friday, 22.11.2019. I know also that 2 other members of drivers team will not be able to be on the meeting so as we will not have quorum on this meeting, lets cancel it. See You on the meeting next week, on 29.11.2019 where (I hope) we should have quorum even if it's just after Thanksgiving. -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[cloudkitty] 18/11 IRC meeting recap
by Luka Peschke 18 Nov '19

18 Nov '19

Hello, This is the recap for today's IRC meeting of the cloudkitty team. The agenda can be found at [1] and the logs can be found at [2]. CloudKitty 11.0.1 ================= CloudKitty 11.0.1 has been released. It includes a fix for security issue on the GET /v1/dataframes and GET /v2/dataframes endpoints which had been introduced during the train development cycle. Various updates =============== Two patches had been discussed during the previous meeting: one for developer documentation for scope fetchers, and one allowing to group results by timestamps on GET /v2/summary. Both have been merged. API improvements ================ Two patches improving the API are currently under review: - The first one updates the way oslo.context and oslo.policy are used. External reviews on this one would be very helpful [3] - The second one improves the way various drivers are loaded in the v2 API. It is available at [4] Standalone cloudkitty dashboard =============================== Julien Pinchelimouroux (julien-pinchelim) has been working on a standalone dashboard (which will also support keystone authentication)for cloudkitty. A 0.1.0 release should happen during in Q4 2019. Some screenshots are available at [5] Cheers, -- Luka Peschke (peschk_l) [1] https://etherpad.openstack.org/p/cloudkitty-meeting-topics [2] http://eavesdrop.openstack.org/meetings/cloudkitty/2019/cloudkitty.2019-11-… [3] https://review.opendev.org/#/c/692333/ [4] https://review.opendev.org/#/c/686393/ [5] https://kutt.it/khA6yF

1 0

[horizon] Changing the release model to cycle-with-intermediary
by Akihiro Motoki 18 Nov '19

18 Nov '19

Hi, I just proposed a patch to change the horizon release model to cycle-with-intermediary. https://review.opendev.org/#/c/694772/ It was discussed during the Shanghai PTG. Horizon provides GUI to users. On the other hand, it is a library for horizon plugins. When horizon plugins would like to use recent changes or to avoid bugs, they need to consume beta releases of horizon. More frequent releases of horizon would make more sense, so I am proposing the release model change. If there are concerns, reply to this thread or drop comments in the review mentioned above. Thanks, Akihiro Motoki (irc: amotoki)

1 0

[neutron] Removing neutron-interconnection out of stadium
by Slawek Kaplonski 17 Nov '19

17 Nov '19

Hi neutrinos, As we discussed during the Shanghai PTG I just proposed changes to move neutron-interconnection project out of stadium to "x/" namespace. So it will not be official neutron project anymore after those changes. Patches for that are in [1] and [2] I also proposed to remove neutron-interconnection api-ref from neutron-lib. Patch is here [3]. Please review it :) [1] https://review.opendev.org/#/c/694478/ [2] https://review.opendev.org/#/c/694480/ [3] https://review.opendev.org/#/c/694466/ -- Slawek Kaplonski Senior software engineer Red Hat

4 5

[oslo][i18n][pbr] get_available_languages() always only returns ['en_US']
by info＠dantalion.nl 16 Nov '19

16 Nov '19

Hello, Across several projects I have noticed that both with unit tests or while the service is running calling olso_i18n.get_available_languages() only returns ['en_US'] (which is inserted as default). Even though the projects I have tested this on have several languages available in the locale directory. Calling any of the python setup.py extract_messages / compile_catalog / update_catalog / install commands does not solve this. However, when I manually copy the .mo files into /usr/share/locale/**/LC_MESSAGES it works as expected. My questions are: Surely there must be a less manual method to properly install all locale files but what is it? Why aren't the locale files installed by pbr when python setup.py install is called? I hope anyone knows the answers to these questions. Kind regards, Corne Lukken (Dantali0n)

1 0

[nova] Displayed state of VM when nova-compute is down/unresponsive.
by Laurent Dumont 16 Nov '19

16 Nov '19

Hey everyone, We had a discussion with some colleagues at work. There was some confusion over the expected behavior of Openstack/Nova regarding the state of VMs on a compute that is down(or one where Nova is in a "bad" state and unable to update properly). Right now, it seems that the VMs will stay in the last state they we're seen. We we're wondering if there was a way to expose the fact that the underlying hypervisor is down? Something like a "Warning : no data from compute since xx:xx:xx" I did not see any documentation regarding a possible configuration option somewhere but a lot of posts with people with similar questions. I understand that the state of the VM shouldn't be changed based on the status of a compute - but exposing the fact that the state itself is not current might be good middle-ground. I do see a possible issue with the fact that the hypervisor itself is not known if the User/Project is not admin. Is anyone aware of anything similar in the past? Thanks!

2 1

Intel 3rd Party CI - Offline until Monday due to upgrades.
by Gresham, Timothy 16 Nov '19

16 Nov '19

Infrastructure upgrades are occurring in the lab which hosts Intel's OpenStack 3rd party CI. These upgrades will require us to take our CI offline for the weekend. Jobs covering the following areas will be offline. * Persistent memory * Tap as a Service * NFV * PCI * SRIOV Jobs covering Cinder/RSD should not be impacted. Service is expected to be restored Monday afternoon Pacific time. We will send out another email once service has been restored. Tim Gresham Cloud Engineer - Intel Corporation Intel Architecture, Graphics, and Software

1 0

[Neutron] OVS forwarding issues
by Volodymyr Litovka 15 Nov '19

15 Nov '19

Dear colleagues, just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted: - from other VMs which reside on the same host (let's name them "local VMs") - to other VMs which reside on other hosts (let's name them "remote VMs") Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there: # ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment. Of course, there is br-tun which connected over vxlan to all other hosts and to br-int: Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} but MAC table on br-tun is empty as well: # ovs-appctl fdb/show br-tun port VLAN MAC Age # Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue. I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue? We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is: @controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true @agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi] Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison

3 4

[kolla] Shutdown ordering of MariaDB containers?
by Eddie Yen 15 Nov '19

15 Nov '19

Hi everyone, I want to ask about the order of shutdown MariaDB (or mean controller) node. For previous steps we found is usually shutdown slaves first, then master [1]. But we found that the MariaDB still get container restarting issue even I followed the step after booting up the cluster. Below is that I did when shutdown/boot up controller. 1. Shutdown the slaves first, then master 2. Boot master first, then slaves. For looking which one is master, we usually looking for the haproxy log and find which mariadb node that the last session access the DB. Or looking for which mariadb container has "--wsrep-new-cluster" in BOOTSTRAP_ARGS. Does anyone has experience about this? Many thanks, Eddie. [1] https://bugs.launchpad.net/kolla-ansible/+bug/1712087

2 2

[oslo] Shanghai Wrapup
by Ben Nemec 15 Nov '19

15 Nov '19

I wrote up a bunch of thoughts about Oslo stuff in Shanghai: http://blog.nemebean.com/content/oslo-shanghai Hopefully I covered everything (and accurately) but if I messed anything up I blame jet lag. :-P -Ben

2 1

Re: [nova][ptg] Flavor explosion
by Brin Zhang(张百林) 15 Nov '19

15 Nov '19

Hi all, Based on the discussion on the Train PTG, and reference to the records on the etherpad and ML, I was updated that SPEC, and I think there are some details need to be discussed, and I have listed some details, if there are any other things that I have not considered, or if some place that I thoughtless, please post a discussion. List some details as follows, and you can review that spec in https://review.opendev.org/#/c/663563. Listed details: - Don't change the model of the flavor in nova code and in the db. - No change for operators who choose not to request the flavor extra specs group. - Requested more than one flavor extra specs groups, if there are different values for the same spec will be raised a 409. - Flavor in request body of server create that has the same spec in the request ``flavor_extra_specs_group``, it will be raised a 409. - When resize an instance, you need to compare the ``flavor_extra_specs_group`` with the spec request spec, otherwise raise a 400. ---------------------------------------------------------------------------------------- Items: [lists.openstack.org代发][nova][ptg] Flavor explosion original spec: https://review.opendev.org/#/c/663563 with -2s The first round of discussion was resulted in no agreement. Then on Friday we revisited the issue based on mdbooth's proposal about composability. Agreement in the room: * Do not try to change the model of the flavor in nova code and in the db. * Define a "ComposableFlavorBit" (bikeshed on the name please) REST API entity that can hold any kind of flavor bits (extra specs, normal flavor fields), propose some format in the spec for it. This entity can only be created by the admin by default * Extend the server create REST API to allow the end user to specify what "ComposableFlavorBit"s she wants to add to the "base" flavor she used in the create request. * The nova api then merges the "ComposableFalvorBit"s with the base flavor and embed the resulted flavor object into the instance. * Do a similar thing for resize TODOs: * brinzhang (with possible help from yawang) to re-write the spec Cheers, gibi

2 2

[nova] Thoughts on exposing exception type to non-admins in instance action event
by Matt Riedemann 15 Nov '19

15 Nov '19

tl;dr: What do people think about storing and showing the *type* of exception that is recorded with a failed instance action event (like a fault) to the owner of the server who may not be an admin? Details: As noted here [1] and recreated here [2] the instance action event details that a non-admin owner of a server sees do not contain any useful information about what caused the failure of the action. Here is an example of a failed resize from that paste (this is what the non-admin owner of the server would see): $ openstack --os-compute-api-version 2.51 server event show vm2 req-11487504-da59-411b-b3b8-267bebe9b0d2 -f json -c events { "events": [ { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "cold_migrate", "result": "Error" }, { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "conductor_migrate_server", "result": "Error" } ] } Super useful, right? In this case scheduling failed for the resize so the instance is not in ERROR status which means the user cannot see a fault message with the NoValidHost error either. The admin can see the traceback in the failed action event list: $ openstack --os-compute-api-version 2.51 server event show 3ef043ea-e2d7-4565-a401-5c758e149f23 req-11487504-da59-411b-b3b8-267bebe9b0d2 -f json -c events { "events": [ { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "traceback": " File \"/opt/stack/nova/nova/conductor/manager.py\", line 301, in migrate_server\n host_list)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 367, in _cold_migrate\n raise exception.NoValidHost(reason=msg)\n", "event": "cold_migrate", "result": "Error" }, { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "traceback": " File \"/opt/stack/nova/nova/compute/utils.py\", line 1411, in decorated_function\n return function(self, context, *args, **kwargs)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 301, in migrate_server\n host_list)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 367, in _cold_migrate\n raise exception.NoValidHost(reason=msg)\n", "event": "conductor_migrate_server", "result": "Error" } ] } So when the admin gets the support ticket they can at least tell that scheduling failed and then dig into why. My idea is to store the exception *type* with the action event, similar to the recorded instance fault message for non-NovaExceptions [3] which will show to the non-admin owner of the server if the server status is ERROR or DELETED [4]. We should record the exc_val to get a prettier message like "No valid host was found." but that could leak details in the error message that we don't want non-admins to see [5]. With what I'm thinking, the non-admin owner of the server could see something like this for a failed event: { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "cold_migrate", "result": "Error", "details": "NoValidHost" } That's pretty simple, doesn't leak details, and at least indicates to the user that maybe they can retry the resize with another flavor or something. It's just an example. This would require a microversion so before writing a spec I wanted to get general feelings about this in the mailing list. I accept that it might not really be worth the effort so that's good feedback if it's how you feel (I'll only cry a little). [1] https://review.opendev.org/#/c/693937/2/nova/objects/instance_action.py [2] http://paste.openstack.org/show/786054/ [3] https://github.com/openstack/nova/blob/20.0.0/nova/compute/utils.py#L101 [4] https://github.com/openstack/nova/blob/20.0.0/nova/api/openstack/compute/vi… [5] https://bugs.launchpad.net/nova/+bug/1851587 -- Thanks, Matt

5 11

Order filters by cost
by Albert Braden 15 Nov '19

15 Nov '19

I'm working on a list of filters ordered by cost. This is what I have so far. Does this look reasonably correct for Rocky? Cheap Filters: AllHostsFilter - does no filtering. It passes all the available hosts. AvailabilityZoneFilter - filters hosts by availability zone. AggregateInstanceExtraSpecsFilter - checks aggregate metadata set with aggregate_instance_extra_specs. All hosts are passed if no extra_specs are specified. AggregateCoreFilter - filters hosts by CPU core number with per-aggregate cpu_allocation_ratio setting. AggregateRamFilter - filters hosts by RAM with per-aggregate ram_allocation_ratio setting. AggregateDiskFilter - filters hosts by disk allocation with per-aggregate disk_allocation_ratio setting. AggregateNumInstancesFilter - filters hosts by number of instances with per-aggregate max_instances_per_host setting. AggregateIoOpsFilter - filters hosts by I/O operations with per-aggregate max_io_ops_per_host setting. AggregateMultiTenancyIsolation - isolate tenants in specific aggregates. AggregateTypeAffinityFilter - limits instance_type by aggregate. AggregateImagePropertiesIsolation - isolates hosts based on image properties and aggregate metadata. DifferentHostFilter - allows the instance on a different host from a set of instances. SameHostFilter - puts the instance on the same host as another instance in a set of instances. ComputeFilter - passes all hosts that are operational and enabled. NumInstancesFilter - filters compute nodes by number of running instances. IoOpsFilter - filters hosts by concurrent I/O operations. More Expensive Filters: ServerGroupAntiAffinityFilter - This filter implements anti-affinity for a server group. ServerGroupAffinityFilter - This filter works the same way as ServerGroupAntiAffinityFilter. The difference is that when you create the server group, you should specify a policy of 'affinity'. ImagePropertiesFilter - filters hosts based on properties defined on the instance's image. Doc on setting image properties is here: https://docs.openstack.org/glance/rocky/admin/useful-image-properties.html IsolatedHostsFilter - filter based on isolated_images, isolated_hosts and restrict_isolated_hosts_to_isolated_images flags. SimpleCIDRAffinityFilter - allows a new instance on a host within the same IP block. MetricsFilter - filters hosts based on metrics weight_setting. Most Expensive Filters: PciPassthroughFilter - Filter that schedules instances on a host if the host has devices to meet the device requests in the 'extra_specs' for the flavor. ComputeCapabilitiesFilter - checks that the capabilities provided by the host compute service satisfy any extra specifications associated with the instance type. NUMATopologyFilter - filters hosts based on the NUMA topology requested by the instance, if any. JsonFilter - allows simple JSON-based grammar for selecting hosts. Deprecated Filters: Please don't use any of these; they are obsolete in Rocky: RetryFilter - filters hosts that have already been attempted for scheduling. Obsolete since Queens. RamFilter - filters hosts by their RAM. Obsolete since Pike. CoreFilter - filters based on CPU core utilization. Obsolete since Pike. DiskFilter - filters hosts by their disk allocation. Obsolete since Pike.

2 1

[all] requirements-check failures
by Sean McGinnis 15 Nov '19

15 Nov '19

Hey everyone, You may have noticed some odd failures with the requirements-check job on your patches lately. The requirements team is aware of this issue and are working to get it resolved ASAP. I believe things should be good again once https://review.opendev.org/694248 lands. So for the time being, please hold off on doing rechecks on these patches. This job should only be running for patches that touch any of the requirements files. As a workaround for now, if your patch can make a change without modifying requirements, that should bypass the need to run this job. Another alternative would be to add: Depends-on: https://review.opendev.org/694248 to your commit message, but hopefully that will not be necessary once this patch makes it through and the test is fixed. Sorry for the inconvenience this has caused. Sean

2 2

[nova][ptg] Resource provider delete at service delete
by Balázs Gibizer 15 Nov '19

15 Nov '19

ML thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html Agreements in the room: * Check ongoing migration and reject the delete if migration with this compute having the source node exists. Let operator confirm the migrations * Cascade delete providers and allocations in placement. * in case of evacuated instances this is the right thing to do * in any other dangling allocation case nova has the final thrut so nova has the authority to delete them. * Document possible ways to reconcile Placement with Nova using heal_allocations and eventually the audit command once it's merged. Cheers, gibi

2 4

[nova][ironic] nova docs bug for ironic looking for an owner
by Matt Riedemann 15 Nov '19

15 Nov '19

While discussing some tribal knowledge about how ironic is the black sheep of nova compute drivers I realized that we (nova) have no docs about the ironic driver like we do for other drivers, so we don't mention anything about the weird cardinality rules around compute service : node : instance and host vs nodename things, how to configure the service for HA mode, how to configure baremetal flavors with custom resource classes, how to partition for conductor groups, how to deal with scaling issues, missing features (migrate), etc. I've opened a bug in case someone wants to get started on some of that information: https://bugs.launchpad.net/nova/+bug/1852446 -- Thanks, Matt

2 1

[tripleo] Adding Alex Schultz as OVB core
by Ben Nemec 15 Nov '19

15 Nov '19

Hi, After a discussion with Wes in Shanghai about how to make me less of a SPOF for OVB, one of the outcomes was that we should try to grow the OVB core team. Alex has been reviewing a lot of the patches to OVB lately and obviously has a good handle on how all of this stuff fits together, so I've added him to the OVB core team. Thanks and congratulations(?) Alex! :-) -Ben

2 1

[nova][oot] Adding `context` to ComputeDriver.unrescue
by Eric Fried 15 Nov '19

15 Nov '19

Though still very much WIP, emulated TPM [1] is looking like it will need to add the RequestContext to the ``unrescue`` ComputeDriver method [2]. This is a heads up to out-of-tree virt driver maintainers to keep an eye on this patch, as you will need to update your overrides accordingly once it merges. Thanks, efried [1] https://review.opendev.org/#/c/631363/ [2] https://review.opendev.org/#/c/631363/31/nova/virt/driver.py

1 0

[neutron] Review priorities
by Slawek Kaplonski 14 Nov '19

14 Nov '19

Hi neutrinos, According to our discussion during Train retrospective in Shanghai, I added "review-priority" label for neutron projects. It can be set by every core team member to values like: -1 - Branch Freeze +1 - Important Change +2 - Gate Blocker Fix / Urgent Change You can use dashboard like [1] to track such high priority patches and review them. I will also add some note about this to our docs this week to make it clear and visible for everyone. [1] https://tinyurl.com/vezk6n6 -- Slawek Kaplonski Senior software engineer Red Hat

3 2

[oslo] Adding Michael Johnson as Taskflow core
by Ben Nemec 14 Nov '19

14 Nov '19

Hi, After discussion with the Oslo team, we (and he) have agreed to add Michael as a Taskflow core. He's done more work on the project than anyone else still active in Oslo and also works on a project that consumes it so he likely understands it better than anyone else at this point. Welcome Michael and thanks for your contributions! -Ben

4 3

[stable][glance] Proposal to add Abhishek Kekane to glance-stable-maint
by Brian Rosmaita 14 Nov '19

14 Nov '19

I'd like to propose adding Abhishek Kekane to the Glance stable maintenance team. He's been a glance core for a few years now, and we are currently understaffed in glance-stable-maint. Plus, he's the current Glance PTL. cheers, brian

3 3

[nova] Today's meeting
by Eric Fried 14 Nov '19

14 Nov '19

Attendance at today's nova meeting was sparse, to say the least. Predictably, some forgot about DST [1], some had conflicts, some are jetlagged, some probably all three. Most hot topics are on ML threads anyway. I and others have updated the meeting agenda [2] with links to those threads. Please be sure to chime in on topics of interest. Thanks, efried [1] DST is stupid and should be abolished [2] https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting

1 0

[Neutron] VPNaaS using certs
by Sriram 14 Nov '19

14 Nov '19

Hi, I would like to know if VPNaaS in openstack provides ipsec with cert based authentication mechanism using certs. Documentation says only psk based authentication is supported. Please advise. Regards, Sriram

1 0

[neutron][ci] Jobs cleaning
by Slawek Kaplonski 14 Nov '19

14 Nov '19

Hi, As we discussed during the PTG, I'm now checking what multinode and singlenode jobs we are exactly running in Neutron CI and what jobs can be potentially removed maybe. Here is what I found +-------------------------------+-----------------------------------------------+-----------------------------------------------------------+ | Singlenode job | Multinode job | Comments | +-------------------------------+-----------------------------------------------+-----------------------------------------------------------+ | neutron-tempest-dvr | neutron-tempest-plugin-dvr-multinode-scenario | Singlenode job runs tempest tests, | | | (non-voting) | multinode job runs tests from neutron-tempest-plugin repo | | | | multinode job isn't stable currently | +-------------------------------+-----------------------------------------------+-----------------------------------------------------------+ | tempest-integrated-networking | tempest-multinode-full-py3 (non-voting) | Singlenode job runs tempest tests related to neutron/nova,| | | | Multinode job runs all tempest tests | | | | multinode job is stable enough to make it voting IMO | +-------------------------------+-----------------------------------------------+-----------------------------------------------------------+ | grenade-py3 | neutron-grenade-multinode | Both jobs runs the same tests and are voting already | +-------------------------------+-----------------------------------------------+-----------------------------------------------------------+ I also found that we have few jobs which we ver similar but the only difference is that one runs tempest tests and other runs tests from neutron tempest plugin. Such jobs are: neutron-tempest-plugin-scenario-openvswitch-iptables_hybrid and neutron-tempest-iptables_hybrid neutron-tempest-plugin-scenario-linuxbridge and neutron-tempest-linuxbridge Do we need all those jobs? Maybe we can simply stay only with neutron-tempest-plugins jobs for those configurations? Or maybe we should "merge" them and run tests from both tempest and neutron-tempest-plugin in one job? -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[tc] Add non-voting py38 for ussuri
by Corey Bryant 14 Nov '19

14 Nov '19

Hello TC members, Python 3.8 is available in Ubuntu Bionic now and while I understand it's too late to enable voting py38 unit tests for ussuri, I'd like to at least enable non-voting py38 unit tests. This email is seeking approval and direction from the TC to move forward with enabling non-voting py38 tests. For some further background: The next release of Ubuntu, Focal (20.04) LTS, is scheduled to release in April 2020. Python 3.8 will be the default in the Focal release, so I'm hopeful that non-voting unit tests will help close some of the gap. I have a review here for the zuul project template enablement for ussuri: https://review.opendev.org/#/c/693401 Also should this be updated considering py38 would be non-voting? https://governance.openstack.org/tc/reference/runtimes/ussuri.html Thanks, Corey

9 17

[neutron][docs] networking-onos EOL?
by Mark Goddard 14 Nov '19

14 Nov '19

Hi, We (kolla) had a bug report [1] from someone trying to use the neutron onos_ml2 ML2 driver for the ONOS SDN controller. As far as I can tell [2], this project hasn't been released since 2015. However, the 'latest' documentation is still accessible [3], and does not mention that the project is dead. What can we do to help steer people away from projects like this? Cheers, Mark [1] https://bugs.launchpad.net/bugs/1850763 [2] https://pypi.org/project/networking-onos/#history [3] https://docs.openstack.org/networking-onos/latest/

3 3

[nova][ptg] pinned and unpinned CPUs in one instance
by Balázs Gibizer 14 Nov '19

14 Nov '19

spec: https://review.opendev.org/668656 Agreements from the PTG: How we will test it: * do functional test with libvirt driver, like the pinned cpu tests we have today * donyd's CI supports nested virt so we can do pinned cpu testing but not realtime. As this CI is still work in progress we should not block on this. * coverage inhttps://opendev.org/x/whitebox-tempest-pluginis a nice to have Naming: use the 'shared' and 'dedicated' terminology Support both the hw:pinvcpus=3 and the resources:PCPU=2 flavor extra specs syntaxtbut not in the same flavor. The resources:PCPU=2 syntax will have less expression power until nova models NUMA in placement. So nova will try to evenly distribute PCPUs between numa nodes. If it not possible we reject the request and ask the user to use the hw:pinvcpus=3 syntax. Realtime mask is an exclusion mask, any vcpus not listed there has to be in the dedicated set of the instance. TODOInvestigate whether we want to enable NUMA by default * Pros: Simpler, everything is NUMA by default * Cons: We'll either have to break/make configurablethe 1:1 guest:host NUMA mapping else we won't be able to boot e.g. a 40 core shared instance on a 40 core, 2 NUMA node host Cheers, gibi

4 6

[nova][ptg] Expose auto converge and post copy
by Balázs Gibizer 14 Nov '19

14 Nov '19

spec: https://review.opendev.org/#/c/687199 There was multiple discussion during the PTG around this. I think at the end mdbooth and yawang found a possible solution that they liked and sounded OK to me too. Unfortunately the etherpad was not updated and my memory is bad enough that I cannot gather what was the exact proposal. yawang: could you please post a short summary here and / or simply update the spec with the discussed solution? Cheers, gibi

3 4

Re: [nova][ptg] Flavor explosion
by Brin Zhang(张百林) 14 Nov '19

14 Nov '19

Thanks Gibizer, I will update this specs to the ussuri directory, and update & reply your comments. brinzhang > 发件人: Balázs Gibizer [mailto:balazs.gibizer@est.tech] > > On Sun, Nov 10, 2019 at 16:09, Brin Zhang(张百林) > <zhangbailin(a)inspur.com> wrote: > > Hi all, > > Based on the discussion on the Train PTG, and reference to the > > records on the etherpad and ML, I was updated that SPEC, and I think > > there are some details need to be discussed, and I have listed some > > details, if there are any other things that I have not considered, or > > if some place that I thoughtless, please post a discussion. > > > > List some details as follows, and you can review that spec in > > https://review.opendev.org/#/c/663563. > > > > Listed details: > > - Don't change the model of the flavor in nova code and in the db. > > > > - No change for operators who choose not to request the flavor extra > > specs group. > > > > - Requested more than one flavor extra specs groups, if there are > > different values for the same spec will be raised a 409. > > > > - Flavor in request body of server create that has the same spec in > > the request ``flavor_extra_specs_group``, it will be raised a 409. > > > > - When resize an instance, you need to compare the > > ``flavor_extra_specs_group`` with the spec request spec, otherwise > > raise a 400. > > > > Thanks Brin for updating the spec, I did a review round on it and left comments. > > gibi >

1 0

[horizon] next weekly meeting cancelled
by Akihiro Motoki 14 Nov '19

14 Nov '19

Hi, The weekly team meeting next week (Nov 20) is cancelled. I will be on a business trip to join a conference in US and cannot run it. We agreed to cancel it in the team meeting yesterday. Akihiro Motoki (amotoki)

1 0

Keystone user ID case sensitivity issue
by PARSONS, CLIFF 14 Nov '19

14 Nov '19

Hello everyone! My organization has a need to make the user name/ID retrieval from Heat template to be case insensitive. For example: suppose we already have a user in keystone, "xyz123". Then we have a client that creates a heat stack containing a UserRoleAssignment resource, in which the user was specified as "XYZ123". The user would not be found in the Keystone database (due to Keystone user IDs being case sensitive) and the role assignment would not occur. Either Keystone could be changed so that its users are treated case insensitive, or we could make the change to heat (Heat KeystoneClientPlugin class) like in https://review.opendev.org/#/c/694117/ so that it converts to lower case before querying keystone. Can I get some thoughts on this? Would something like this be acceptable at all? Would we need to make it configurable, and if we did, would that be acceptable? Thanks in advance for your thoughts/concerns/suggestions. Thank you, Cliff Parsons

3 2

[nova][ptg] Allow compute nodes to use DISK_GB from shared storage RP by using aggregate relationship
by Balázs Gibizer 14 Nov '19

14 Nov '19

spec: https://review.opendev.org/#/c/650188/ Agreements from the room[1]: * new config option 'using_shared_disk_provider' (just an example name) on the compute level to ignore the DISK_GB reporting from the driver * new config option 'sharing_disk_aggregate' (just an example name) on the compute level to tell nova compute what is the UUID of the placement aggregate that contains the sharing DISK_GB providers in placement. * the "using_shared_disk_provider" flag necessarly has to be explicit since if not, it would be a chicken-and-egg problem on a greenfields install as the shared RP wouldn't be created * deployer needs to create the sharing disk RP and report inventory / traits on it * deployer needs to define the placement aggregate and add the sharing disk RP into it * when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape * os-hypervisor API response (in a new microversion) will have a link to the sharing disk RP if the compute is so configured. TODO: * tpatil to update the spec [1] https://etherpad.openstack.org/p/nova-shanghai-ptg

4 8

[nova] track error migrations and orphans in Resource Tracker
by Zhong, Luyao 14 Nov '19

14 Nov '19

Hi Nova experts, "Not tracking error migrations and orphans in RT." is probably a bug. This may trigger some problems in update_available_resources in RT at the moment. That is some orphans or error migrations are using cpus/memory/disk etc, but we don't take these usage into consideration. And instance.resources is introduced from Train used to contain specific resources, we also track assigned specific resources in RT based on tracked migrations and instances. So this bug will also affect the specific resources tracking. I draft an doc to clarify this bug and possible solutions: https://etherpad.openstack.org/p/track-err-migr-and-orphans-in-RT Looking forward to suggestions from you. Thanks in advance. Best Regards, Luyao

6 7

Filter costs / filter order
by Albert Braden 14 Nov '19

14 Nov '19

I'm running Rocky and trying to figure out filter order. I'm reading this doc: https://docs.openstack.org/nova/rocky/user/filter-scheduler.html It says: Each filter selects hosts in a different way and has different costs. The order of filter_scheduler.enabled_filters affects scheduling performance. The general suggestion is to filter out invalid hosts as soon as possible to avoid unnecessary costs. We can sort filter_scheduler.enabled_filters items by their costs in reverse order. For example, ComputeFilter is better before any resource calculating filters like RamFilter, CoreFilter. Is there a document that specifies filter costs, or ranks filters by cost? Is there a well-known process for determining the optimal filter order?

2 2

[ironic][ptg] Summary of discussions/happenings related to ironic
by Julia Kreger 14 Nov '19

14 Nov '19

Overall, There was quite a bit of interest in Ironic. We had great attendance for the Project Update, Rico Lin’s Heat/Ironic integration presentation, demonstration of dhcp-less virtual media boot, and the forum discussion on snapshot support for bare metal machines, and more! We also learned there are some very large bare metal clouds in China, even larger than the clouds we typically talk about when we discuss scale issues. As such, I think it would behoove the ironic community and OpenStack in general to be mindful of hyper-scale. These are not clouds with 100s of compute nodes, but with baremetal clouds containing thousands to tens of thousands of physical bare metal machines. So in no particular order, below is an overview of the sessions, discussions, and commentary with additional status where applicable. My apologies now since this is over 4,000 words in length. Project Update =========== The project update was fairly quick. I’ll try and record a video of it sometime this week or next and post it online. Essentially Ironic’s code addition/deletion levels are relatively stable cycle to cycle. Our developer and Ironic operator commit contribution levels have increased in Train over Stein, while the overall pool of contributors has continued to decline cycle after cycle, although not dramatically. I think the takeaway from this is that as ironic has become more and more stable, and that the problems being solved in many cases are operator specific needs or wants, or bug fixes in cases that are only raised in particular environment configurations. The only real question that came out of the project update was, if my memory is correct, was “What does Metal^3 mean for Ironic”, and “Who is driving forward Metal^3?” The answers are fairly straight forward, more ironic users and more use cases from Metal^3 driving ironic to deploy machines. As for who is driving it forward, it is largely being driven forward by Red Hat along with interested communities and hardware vendors. Quick, Solid, and Automatic OpenStack Bare-Metal Orchestration ================================================== Rico Lin, the Heat PTL, proposed this talk promoting the possibility of using ironic naively to deploy bare metal nodes. Specifically where configuration pass-through can’t be made generic or somehow articulated through the compute API. Cases where they may be is where someone wishes to utilize something like our “ramdisk” deploy_interface which does not deploy an image to the actual physical disk. The only real question that I seem to remember coming up was the question why might someone want or need to do this, which again becomes more of a question of doing things that are not quite “compute” API-ish. The patches are available in gerrit[10]. Operator Feedback Session ===================== The operator feedback[0] session was not as well populated with maybe ~20-25 people present. Overall the feeling of the room was that “everything works”, however there is a need and desire for information and additional capabilities. * Detailed driver support matrix * Reduce the deployment times further * Disk key rotation is an ask from operators for drives that claim smart erase support but end up doing a drive wipe instead. In essence, to reduce the overall time spent cleaning * Software RAID is needed at deploy time. * IPA needs improved error handling. - This may be a case where something of the communication flow changes that had been previously discussed could help in that we could actively try and keep track of the agent a little more. Additional discussion will definitely be required. * There does still seem to be some interest in graphical console support. A contributor has been revising patches, but I think it would really help for a vendor to become involved here and support accessing their graphical interface through such a method. * Information and an information sharing location is needed. I’ve reached out to the Foundation staff regarding the Bare Metal Logo Program to see if we can find a common place that we can build/foster moving forward. In this topic, the one major pain point began being stressed, issues with the resource tracker at 3,500 bare metal nodes. Privately another operator reached out with the same issue in the scale of tens of thousands of bare metal nodes. As such, this became a topic during the PTG which gained further discussion. I’ll cover that later. Ironic – Snapshots? =============== As a result of some public discussion of adding snapshot capability, I proposed a forum session to discuss the topic[1] such that requirements can be identified and the discussion can continue over the next cycle. I didn't expect the number of attendees present to swell from the operator's feedback session. The discussion of requirements went back and forth to ultimately define "what is a snapshot" in this case, and "what should Ironic do?" There was quite a bit of interaction in this session and the consensus seemed to be the following: * Don’t make it reliant on nova, for standalone users may want/need to use it. * This could be a very powerful feature as an operator could ``adopt`` a machine into ironic and then ``snapshot`` it to capture the disk contents. * Block level only and we can’t forget about capturing/storing content checksum * Capture the machine’s contents with the same expectation as we would have for a VM, and upload this to someplace. In order to make this happen in a fashion which will scale, the ironic team will likely need to leverage the application credentials. Ironically reeling in large bare metal deployment without PXE ============================================== This was a talk submitted by Ilya Etingof, who unfortunately was unable to make it to the summit. Special thanks goes to Both Ilya and Richard Pioso for working together to make this demonstration happen. The idea was to demonstrate where the ironic team sees the future of deployment of machines on the edge using virtual media and how vendors would likely interact with that in some cases as slightly different mechanics may be required even if the BMCs all speak Redfish, which is the case for a Dell iDRAC BMC. The idea[2] ultimately being is that the conductor would inject the configuration information into the virtual media ISO image that is attached via virtual media negating the need for DHCP. We have videos posted that allow those interested to see what this functionality looks like with neutron[3] and without neutron[4]. While the large audience was impressed, it seemed to be a general surprise that Ironic had virtual media support in some of the drivers previously. This talk spurred quite a bit of conversation and hallway track style discussion after the presentation concluded which is always an excellent sign. Project Teams Gathering =================== The ironic community PTG attendance was nothing short of excellent. Thank you everyone who attended! At one point we had fifteen people and a chair had to be pulled up to our table for a 16th person to join us. At which point, we may have captured another table and created confusion. We did things a little differently this time around. Given some of the unknowns, we did not create a strict schedule around the topics. We simply went through and prioritized topics and tried to discuss them each as thoroughly as possible until we had reached the conclusion or a consensus on the topic. Topics and a few words on each topic we discussed in the notes section on the PTG etherpad[5]. On-boarding ----------------- We had three contributors that attended a fairly brief on-boarding overview of Ironic. Two of them were more developer focused where as the third was more of an operator focus looking to leverage ironic and see how they can contribute back to the community. BareMetal SIG - Next Steps ------------------------------------- Arne Wiebalck and I both provided an update including current conversations where we saw the SIG, the Logo Program, the white paper, and what should the SIG do beyond the whitepaper. To start with the Logo program, it largely seems there that somewhere along the way a message or document got lost and that largely impacted the Logo Program -> SIG feedback mechanism. I’m working with the OpenStack Foundation to fix that and get communication going again. Largely what spurred that was that some vendors expressed interest in joining, and wanted additional information. As for the white paper, contributions are welcome and progress is being made again. >From a next steps standpoint, the question was raised how do we build up an improved Operator point of contact. There was some consensus that we as a community should try to encourage at least one contributor to attend the operations mid-cycles. This allows for a somewhat shorter feedback look with a different audience. We also discussed knowledge sharing, or how to improve it. Included with this is how do we share best practices. I’ve got the question out to folks at the foundation if there is a better way as part of the Logo program, or if we should just use the Wiki. I think this will be an open discussion topic in the coming weeks. The final question that came up as part of the SIG is how to show activity. I reached out to Amy on the UC regarding this, and it seems the process is largely just reach out to the current leaders of the SIG, so it is critical that we keep that up to date moving forward. Sensor Data/Metrics --------------------------- The barrier between Tenant level information and Operator level information is difficult with this topic. The consensus among the group was that the capability to collect some level of OOB sensor data should be present on all drivers, but there is also a recognition that this comes at a cost and possible performance impact. Mainly this performance impact question was raised with Redfish because this data is scattered around the API where multiple API calls are required, and may even cause some interruption to actively inquire upon some data points. The middle ground in the discussion came to adding a capability of somehow saying “collect power status, temp every minute, fan speeds every five minutes, drive/cpu health data maybe every 30 minutes”. I would be remiss if I didn't note that there was joking about how this would in essence be re-implementation of Cron. What this would end up looking like, we don’t know, but it would provide operators the data resolution necessary for the failure risk/impact. The analogy used was that “If the temperature sensor has risen to an alarm level, either a AC failure or a thermal hot spot forming based upon load in the data center, checking the sensor too often is just not going to result in a human investigating that on the data center floor any faster.” Mainly I believe this discussion largely stresses that the information is for the operator of the bare metal and not to provide insight into a tenant monitoring system, that those activities should largely be done with-in the operating system. One question among the group was if anyone was using the metrics framework built into ironic already for metrics of ironic itself, to see if we can re-use it. Well, it uses a plugin interface! In any event, I've sent a post to the openstack-discuss mailing list seeking usage information. Node Retirement ----------------------- This is a returning discussion from the last PTG, and in discussing the topic we figured out where the discussion became derailed at previously. In essence, the desire was to mix this with the concept of being able to take a node “out of service”. Except, taking a node out of service is an immediate state related flag, where as retiring might be as soon as the current tenant vacates the machine… possibly in three to six months. In other words, one is “do something or nothing now”, and the other is “do something later when a particular state boundary is crossed”. Trying to make one solution for both, doesn’t exactly work. Unanimous consensus among those present was that in order to provide node retirement functionality, that the logic should be similar to maintenance/maintenance reason. A top level field in the node object that would allow API queries for nodes slated for retirement, which helps solve an operator workflow conundrum “How do I know what is slated for retirement but not yet vacated?” Going back to the “out of service” discussion, we reached consensus that this was in essence a “user declarable failed state”, and as such that it should be done only in the state machine as it is in the present, not a future action. Should we implement out of service, we’ll need to check the nova.virt.ironic code and related virt code to properly handle nodes dropping from `ACTIVE` state, which could also be problematic and need to be API version guarded to prevent machines from accidentally entering `ERROR` state if they are not automatically recovered in nova. Multi-tenancy ------------------ Lots of interest existed around making the API somewhat of a mutli-tenant aware interaction, and the exact interactions and uses involved there are not exactly clear. What IS clear is that providing functionality as such will allow operators to remove complication in their resource classes and tenant specific flavors which is presently being used to enable tenant specific hardware pools. The added benefit of providing some level for normally non-admin users to access the ironic API is that it would allow those tenants to have a clear understanding of their used resources and available resources by directly asking ironic, where as presently, they don’t have a good way to collect nor understand that short of asking the cloud operator when it comes to bare metal. Initial work has been posted for this to gerrit[6]. In terms of how tenants resources would be shared, there was consensus that the community should stress that new special use tenants should be created for collaborative efforts. There was some discussion regarding explicitly dropping fields for non-privileged users that can see the nodes, such as driver_info and possibly even driver_internal_info. Definitely a topic that requires more discussion, but that would solve operator reporting and use headaches. Manual Cleaning Out-Of-Band ---------------------------------------- The point was raised that we unconditionally start the agent ramdisk to perform manual cleaning. Except, we should support a method of out of band cleaning operators to only be executed so the bare metal node doesn’t need to be booted to a ramdisk. The consensus seemed to be that we should consider a decorator or existing decorator change that allows the conductor to hold off actually powering the node on for ramdisk boot unless or until a step is reached that is not purely out of band. In essence, fixing this allows a “fix_bmc” out of band clean step to be executed first without trying to modify BMC settings, which would presently fail. Scale issues ----------------- A number of scaling issues between how nova and ironic interact, specifically with the resource tracker and how inventory is updated from ironic and loaded into nova. Largely this issue revolves around the concept in nova that each ``nova-compute`` is a hypervisor. And while one can run multiple ``nova-compute`` processes to serve as the connection to ironic, the underlying lock in Nova is at the level of the compute node, not the node level. This means as thousands of records are downloaded, synced, copied into the resource tracker, the compute process is essentially blocked from other actions while this serialized job runs. In a typical VM case, you may only have at most a couple hundred VMs on a hypervisor, where as with bare metal, we’re potentially servicing thousands of physical machines. It should be noted that there are several large scale operators that indicated during the PTG that this was their pain point. Some of the contributors from CERN sat down with us and the nova team to try and hammer out a solution to this issue. A summary of that cross project session can be found at line 212 in the PTG etherpad[0]. But there is another pain point that contributes to this performance issue and that is the speed at which records are returned by our API. We’ve had some operators voice some frustration with this before, and we should at least be mindful of this and hopefully see if we can improve record retrieval performance. In addition to this, if we supported some form of bulk “GET” of nodes, it might be able to be leveraged as opposed to a get on each node one at a time which is presently what occurs in the nova-compute process. Boot Mode Config ------------------------ Previously, when scheduling occurred with flavors and filters were appropriately set, if a machine was declared as supporting only one boot mode, requests would only ever land on that node. Now with Traits, this is a bit different and unfortunately optional without logic to really guard the setting application for an instance. So in this case, if filters are such that a request for a Legacy boot instance lands on a UEFI only machine, we’ll still try to deploy it. In reality, we really should try and fail fast. Ideally the solution here is we consult with the BMC through some sort of get_supported_boot_modes method, and if we determine a mismatch between what the settings are or what the requested instance is from the data we have, we fail the deploy. This ultimately may require work in the nova.virt.ironic driver code to identify the cause of the failure being an invalid configuration and reporting that back, however it may not be fatal on another machine. Security of /heartbeat and /lookup endpoints ----------------------------------------------------------- We had a discussion of adding some additional layers of security mechanics around the /heartbeat and /lookup endpoints in ironic’s REST API. These limited endpoints are documented as being unauthenticated, so naturally some issues can arise from these and we want to minimize the vectors in which an attacker that has gained access to a cleaning/provisioning/rescue network could possibly impersonate a running ironic-python-agent. Conversely, the ironic-python-agent runs in a similar fashion, intended to run on secure trusted networks which is only accessible to the ironic-conductor. As such, we also want to add some validation to the API request is from the same Ironic deployment that IPA is heart-beating to. The solution to this introduce a limited lifetime token that is unique per node per deployment. It would be stored in RAM on the agent, and in the node.driver_internal_info so it is available to the conductor. It would be provided only once via out of band OR via the first “lookup” of a node, and then only become accessible again during known reboot steps. Conceptually the introduction of tokens was well supported in the discussions and there were zero objections to doing so. Some initial patches[7][8] are under development to move this forward. An additional item is to add IP address filtering capabilities to both endpoints such that we only process the heartbeat/lookup address if we know it came from the correct IP address. An operator has written this feature downstream and consensus was unanimous at the PTG that we should accept this feature upstream. We should expect a patch for this functionality to be posted soon. Persistent Agents ------------------------ The use case behind persistent agents is “I want to kexec my way to the agent ramdisk, or the next operating system.” and “I want to have up to date inspection data.” We’ve already somewhat solved the latter, but the former is a harder problem requiring the previously mentioned endpoint security enhancements to be in-place first. There is some interest from CERN and some other large scale operators. In other words, we should expect more of this from an bare metal fleet operations point of view for some environments as we move forward. “Managing hardware the Ironic way” ------------------------------------------------- The question that spurred this discussion was “How do I provide a way for my hardware manager to know what it might need to do by default.” Except, those defaults may differ between racks that serve different purposes. “Rack 1, node0” may need a port set to FiberChannel mode, where as “Rack2, node1” may require it to be Ethernet. This quickly also reaches the discussion of “What if I need different firmware versions by default?” This topic quickly evolved from there and the idea that surfaced was that we introduce a new field on the node object for the storage of such data. Something like ``node.default_config``, where it would be a dictionary sort of like what a user provides for cleaning steps or deploy steps, that provides argument values which is iterated through when in automated cleaning mode to allow operators to fill in configuration requirement gaps for hardware managers. Interestingly enough, even today we just had someone ask a similar question in IRC. This should ultimately be usable to assert desired/default firmware from an administrative point of view. Adrianc (Mellanox) is going to reach out to bdobb (DMTF) regarding the redfish PLDM firmware update interface to see where this may go from here. Edge computing working group session ---------------------------------------------------- The edge working group largely became a session to update everyone on where Ironic was going and where we see things going in terms of managing bare metal at the edge/far-edge. This included some in-depth questions about dhcp-less deployment and related mechanics as well as HTTPBoot’ing machines. Supporting HTTPBoot does definitely seem to be of interest to a number of people, although at least after sharing my context only five or six people in attendance really seemed interested in ironic prioritizing such functionality. The primary blocker, for those that are unaware, is pre-built UEFI images for us to do integration testing for IPv4 HTTPBoot. Functionally ironic already supports IPv6 HTTPBoot via DHCPv6 as part of our IPv6 support with PXE/iPXE, however we also don’t have an integration test job for this code path for the same reason, pre-built UEFI firmware images lack the built-in support. More minor PTG topics ------------------------------- * Smartnics - A desire to attach virtual ports in ironic baremetal nodes with smartnics was raised. Seems that we don’t need to try and create a port entry in ironic. It seems we only need to track/signal and remove the “vif” attachment” to the node in general as there is no physical mac required for that virtual port in ironic. The constraint that at least one MAC address would be required to identify the machine is understood. If anyone sees an issue with this, please raise this to adrianc. * Metal^3 - Within the group attending the PTG, there was not much interest in Metal^3 or using CRDs to manage bare metal resources with ironic hidden behind the CRD. One factor related to this is the desire to define more data to be passed through to ironic which is not presently supported in the CRD definition.. Stable Backports with Ironic's release model ================================== I was pulled into a discussion with the TC and the Stable team regarding frustrations that have been expressed with-in the ironic team regarding stable back-porting of fixes, mainly drivers. There is consensus that it is okay for us as the ironic team to backport drivery things when needed to support vendors as long as they are not breaking or overall behavior contracts. This quickly leads us to needing to also modify constraints for drivery things as well. Constraints changes will continue to be evaluated on a case by case basis, but the general consensus is there is full support to "do the right thing" for ironic's users, vendors, and community. The key is making sure we are on the same page and agreeing to what that right thing is. This is where asynchronous communication can get us into trouble, and I would highly encourage trying to start higher bandwidth discussion when these cases arise in the future. The key takeaway that we should likely keep in mind is policy is there for good reasons, but policy is not and can not be a crutch to prevent the right thing from being done. Additional items worth noting - Q1 Gatherings =================================== There will be an operations mid-cycle at Bloomberg in London, January 7th-8th, 2020. It would be good if at least one ironic contributor could attend as the operators group tends to be closer to the physical baremetal, and it is a good chance to build mutual context between developers and operations people actually using our software. Additionally, we want to gauge the interest of having an ironic mid-cycle in central Europe in Q1 of 2020. We need to identify the number of contributors that would be interested in and able to attend since the next PTG will be in June. Please email me off-list if your interested in attending and I'll make a note of it as we're still having initial discussions. And now I've reached a buffer under-run on words. If there are any questions, just reply to the list. -Julia Links: [0]: https://etherpad.openstack.org/p/PVG-ironic-operator-feedback [1]: https://etherpad.openstack.org/p/PVG-ironic-snapshot-support [2]: https://review.opendev.org/#/c/672780/ [3]: https://tinyurl.com/vwts36l [4]: https://tinyurl.com/st6azrw [5]: https://etherpad.openstack.org/p/PVG-Ironic-Planning [6]: https://review.opendev.org/#/c/689551/ [7]: https://review.opendev.org/692609 [8]: https://review.opendev.org/692614 [9]: https://etherpad.openstack.org/p/ops-meetup-1st-2020 [10]: https://review.opendev.org/#/q/topic:story/2006403+(status:open+OR+status:m…

1 1

[cinder] virtual PTG datetime selection poll
by Brian Rosmaita 13 Nov '19

13 Nov '19

At the Shanghai PTG last week, the Cinder team decided to hold a Virtual PTG the last week of November. The format will be 2 two-hour sessions, ideally held on consecutive days. Since most people have the 16:00 UTC Cinder meeting already on their calendars, I suggest that we skip the meeting on 27 November and instead use that time for Virtual PTG. So, basically, I'd like to meet: Wednesday for sure, and either Tuesday or Thursday (with Monday or Friday as possibilities if Tuesday or Thursday are impossible for too many people). Let me know what your preferences are on this poll: https://forms.gle/rKDJpSZvAxbnBESp7 The poll closes at 23:39 on *Tuesday* 19 November 2019. thanks, brian

1 1

[cinder] meeting time change poll
by Brian Rosmaita 13 Nov '19

13 Nov '19

The poll to help decide whether we move the time of the weekly Cinder meeting is now available: https://forms.gle/kA2JGzoxegy2KRDB6 The poll closes on 20 November at 23:59 UTC. If the time is changed, it will be effective for the first meeting in December (4 December 2019). cheers, brian

1 1

[ironic] Anyone using the process metrics collection functionality in ironic?
by Julia Kreger 13 Nov '19

13 Nov '19

A question from the PTG was raised if anyone was using the existing statsd metrics publishing support in ironic/ironic-python-agent to publish internal performance times? There seems to be some interest in expanding this metric data publishing capability so Prometheus can also be used, but before we really even think of heading down that path, we wanted to understand if there were present users of that feature. Thanks, -Julia

1 0

[blazar] Why/how does Blazar use Keystone trusts?
by Jason Anderson 13 Nov '19

13 Nov '19

Hi Blazar contributors, We hit an issue today involving trusts in Blazar, where a host couldn't be deleted due to some issue authenticating against the trust associated with the host. We still haven't resolved this issue, but it felt odd to me: why is a trust even involved here? I have often wondered what the reason is for using trusts in Blazar, as I can't think of anything Blazar is doing that could not be done by the Blazar system user (and in fact, many operations are done via this user... via another trust.) There are also issues where a user leaves a project before their leases have ended; in this case Blazar has difficulty cleaning up because it tries to resurrect a trust that is not tied to a valid user/project relationship. Does anybody have context on to how trusts are used in Blazar and if they are still necessary? Does it make sense to remove this functionality? Thank you, -- Jason Anderson Chameleon DevOps Lead Consortium for Advanced Science and Engineering, The University of Chicago Mathematics & Computer Science Division, Argonne National Laboratory

3 3

[cinder] meeting time reminder
by Brian Rosmaita 13 Nov '19

13 Nov '19

For people who were on Daylight Savings Time but now are not, just a reminder that this week's Cinder meeting at 16:00 UTC may be an hour earlier for you. The meeting-time-change-poll link will be in a separate email. If the time is changed, it will be effective for the first meeting in December (4 December 2019). That's because we'll be having the Virtual PTG during the last week of November (that poll will be out shortly). cheers, brian

1 0

[stable][glance] Proposal to remove Flavio Percoco from glance-stable-maint
by Brian Rosmaita 13 Nov '19

13 Nov '19

I just noticed that Flavio is still a member of glance-stable-maint. Nothing against him personally -- he's an excellent dude -- but he hasn't been working on Glance (or OpenStack) for quite a while now and is no longer a member of glance-core, so he probably shouldn't be on the stable-maint team. (Not that he'd do anything bad, it just makes the glance-stable-maint team look larger than it actually is.) On the off chance he'll see this message, I'd like to thank Flavio for all his hard work in the past keeping Glance stable! cheers, brian

3 3

[ptg] Auto-generated etherpad links !
by Thierry Carrez 13 Nov '19

13 Nov '19

Hi everyone, The PTGbot grew a new feature over the summer. It now dynamically generates the list of PTG track etherpads. You can find that list at: http://ptg.openstack.org/etherpads.html If you haven't created your etherpad already, just follow the link there to create your etherpad. If you have created your track etherpad already under a different name, you can overload the automatically-generated name using the PTGbot. Just join the #openstack-ptg channel and (as a Freenode authenticated user) send the following command: #TRACKNAME etherpad <URL for your etherpad> Example: #keystone etherpad https://etherpad.openstack.org/p/awesome-keystone-pad That will update the link on that page automatically. Hoping to see you in Shanghai! -- Thierry Carrez (ttx)

2 1

[kuryr] [tc] kuryr project mission
by Michał Dulko 13 Nov '19

13 Nov '19

Hi, It's been more than a year after fuxi [1] and fuxi-kubernetes [2] were declared as retired. First one was supposed to integrate Docker with Cinder and Manila. The second was aiming to do the same with Kubernetes. I'm not really sure how to serve the first use case, but it seems like there are some alternatives [3], [4]. fuxi-kubernetes use case is now served by Cloud Provider OpenStack in Cinder [5] and Manila [6] CSI plugins, which seem like a much better fit. Currently the only maintained deliverables in kuryr project are kuryr, kuryr-libnetwork and kuryr-kubernetes. All of them are related to networking. Given that I'd like to propose rephrasing Kuryr mission statement from: > Bridge between container framework networking and storage models > to OpenStack networking and storage abstractions. to > Bridge between container framework networking models > to OpenStack networking abstractions. effectively getting storage out of project scope. Are there any thoughts or objections? Maybe someone sees a better phrasing? Thanks, Michał [1] https://opendev.org/openstack/fuxi [2] https://opendev.org/openstack/fuxi-kubernetes [3] https://rexray.readthedocs.io/en/stable/user-guide/schedulers/docker/plug-i… [4] https://github.com/j-griffith/cinder-docker-driver [5] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/usi… [6] https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/usi…

5 4

[tc][horizon][all] Horizon plugins maintenance
by Ivan Kolodyazhny 13 Nov '19

13 Nov '19

Hi team, As you may know, we've got a pretty big list of Horizon Plugins [1]. Unfortunately, not all of them are in active development due to the lack of resources in projects teams. As a Horizon team, we understand all the reasons, and we're doing our best to help other teams to maintain plugins. That's why we're proposing our help to maintain horizon plugins. We raised this topic during the last Horizon weekly meeting [2] and we'll have some discussion during the PTG [3] too. There are a lot of Horizon changes which affect plugins and horizon team is ready to help: - new Django versions - dependencies updates - Horizon API changes - etc. To get faster fixes in, it would be good to have +2 permissions for the horizon-core team for each plugin. We helped Heat team during the last cycle adding horizon-core to the heat-dashboard-core team. Also, we've got +2 on other plugins via global project config [4] and via Gerrit configuration for (neutron-*aas-dashboard, tuskar-ui). Vitrage PTL agreed to do the same for vitrage-dashboard during the last meeting [5]. Of course, it's up to each project to maintain horizon plugins and it's responsibilities but I would like to raise this topic to the TC too. I really sure, that it will speed up some critical fixes for Horizon plugins and makes users and operators experience better. [1] https://docs.openstack.org/horizon/latest/install/plugin-registry.html [2] http://eavesdrop.openstack.org/meetings/horizon/2019/horizon.2019-10-16-15.… [3] https://etherpad.openstack.org/p/horizon-u-ptg [4] http://codesearch.openstack.org/?q=horizon-core&i=nope&files=&repos=opensta… [5] http://eavesdrop.openstack.org/meetings/vitrage/2019/vitrage.2019-10-23-08.… Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

9 9

[ptg][glance] PTG summary
by Abhishek Kekane 13 Nov '19

13 Nov '19

Hi All, I attended OpenInfra summit and PTG at Shanghai last week. It was an interesting event with lots of discussion happening around different OpenStack projects. I was mostly associated with Glance and cross-projects work related to Glance. There were other topics around Edge, UI and QA. During summit Me and Erno gave a Glance project update where we discussed what we achieved in Train cycle and what we are going to do in Ussuri cycle. As multiple stores feature is stabilized during Train, In Ussuri main focus of glance is on enhancing /v2/import API to import single image into multiple stores and copying existing images to multiple stores to avoid the manual efforts required by operator to copy the image across the stores. New delete API will also be added to delete the image from single store, also cinder driver of glance_store needs refactoring so that it can use multiple backends configured by cinder. Efforts will be continued for cluster awareness of glance API during this cycle as well. Apart from these edge related work, Glance team will also work on removing deprecated registry and related functional tests, removing of sheepdog driver from glance_store, adding s3 driver with multiple stores support in glance_store and some urgent bug fixes. Cross-Project work: In this PTG we had discussion with Nova and Cinder regarding the adoption of multiple store feature of Glance. As per discussion we have finalized the design and Glance team will work together with Nova and Cinder towards adding multiple store support feature in Train cycle. Support for Glance multiple stores in Cinder: As per discussion, volume-type will be used to add which store the image will be uploaded on upload-to-image operation, also cinder will send base image id to glance as a header using which glance will upload the image created from volume to all those stores in which base image is present. Nova snapshots to dedicated store: Agreement is, Nova will send us a base image id to glance as a header using which glance will upload the instance snapshot to all those stores in which base image is present. Talk with QA team: Glance has also talked with QA team for adding new tempest coverage for newly added features in the last couple of cycles, Glance team will work with tempest to add below new tempest tests. 1. New import workflow (image conversion, inject metadata etc.) - Depends on https://review.opendev.org/#/c/545483/ devstack patch 2. Hide old images 3. Multiple stores: https://review.opendev.org/#/c/689104/ in devstack 3.1 Devstack patch + zuul job to setup multiple stores and the job will run on glance and run glance api and scenario tests 4. Delete barbican secrets from glance images 4.1 add the tests the in barbican-tempest-plugin 4.2 run as part of barbican gate using their job 4.3 run that tests with new job (multi stores) on glance gate. do not run barbican job on glance. Below is the Ussuri cycle planning and deadline for Glance. Ussuri milestone planning: Ussuri U1 - December 09-13: 1. Import image in multiple stores (Specs + Implementation) 2. Copy existing image in multiple stores (Specs + Implementation) 3. S3 driver for glance 4. remove sheepdog driver from glance_store 5. Fix subunit parser error 6. Modify existing nova and cinder specs Ussuri U2 - February 10-14 1. Cluster awareness of glance API nodes 2. remove registry code 3. Delete image from single store 4. Nova and Cinder upload snapshot/volume to glance 5. image-import.conf parsing issue with uwsgi Ussuri U3 - April 06-10 1. Multiple cinder store support in glance_store (specs + implementation) 2. Creating image from volume using ceph (slow uploading issue) 3. Image encryption 4. Tempest work Glance PTG planning etherpad: https://etherpad.openstack.org/p/Glance-Ussuri-PTG-planning Let me know if you guys need more details on this. Thanks & Best Regards, Abhishek Kekane

1 0

[nova][ops][stable] Any interest in backporting --dry-run and/or --instance options for heal_allocations?
by Matt Riedemann 13 Nov '19

13 Nov '19

I was helping someone recover from a stuck live migration today where the migration record was stuck in pre-migrating status and somehow the request never hit the compute or was lost. The guest was stopped on the guest and basically the live migration either never started or never completed properly (maybe rabbit dropped the request or the compute service was restarted, I don't know). I instructed them to update the database to set the migration record status to 'error' and hard reboot the instance to get it running again. Then they pointed out they were seeing this in the compute logs: "There are allocations remaining against the source host that might need to be removed" That's because the source node allocations are still tracked in placement by the migration record and the dest node allocations are tracked by the instance. Cleaning that up is non-trivial. I have a troubleshooting doc started for manually cleaning up that kind of stuff here [1] but ultimately just told them to delete the allocations in placement for both the migration and the instance and then run the heal_allocations command to recreate the allocations for the instance. Since this person's nova deployment was running Stein, they don't have the --dry-run [2] or --instance [3] options for the heal_allocations command. This isn't a huge problem but it does mean they could be healing allocations for instances they didn't expect. They could work around this by installing nova from train or master in a VM/container/virtual environment and running it against the stein setup, but that's maybe more work than they want to do. The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally. [1] https://review.opendev.org/#/c/691427/ [2] https://review.opendev.org/#/c/651932/ [3] https://review.opendev.org/#/c/651945/ -- Thanks, Matt

5 5

[nova][ptg] PCI refactoring needs and a strawman proposal inside
by Sylvain Bauza 13 Nov '19

13 Nov '19

Based on some Forum discussions, we had a de facto conversation at the Nova PTG about any potential things we could do for helping our operators, but also potentially Cyborg since they use the PCI passthrough capabilities. The feedback from ops (thanks mnaser) was that PCI passthrough works pretty smoothly but there are some ugly issues where we could improve the UX. The agreement in the room was, at least for Ussuri, to start collecting some ideas on how we could model Placement usage for PCI devices and also write motivations for such things in a spec. We also agreed on a smooth upgrade plan where PCITracker would still be present for a couple of releases until we're able to close the feature gap. Bauzas volunteered for drafting the spec and gibi, stephenfin and bauzas started to sketch up the Placement modeling on a whiteboard, picture to be shared in a follow-up.

3 2

[nova] spec is ready to review and for feature liasion needed
by Brin Zhang(张百林) 13 Nov '19

13 Nov '19

Hi everyone: This spec comment by melwitt, and already updated, wait for review and need a feature liaison. Proposal for a safer noVNC console with password authentication https://review.opendev.org/#/c/623120 Brin Zhang

2 2

[stable][cinder] Proposal to remove John Griffith from cinder-stable-maint
by Brian Rosmaita 13 Nov '19

13 Nov '19

John Griffith has taken on other commitments and stepped down as a cinder-core recently, so it doesn't make sense for him to continue on the cinder-stable-maint list. I'd like to acknowledge his role as "The Father of Cinder", though, and express my thanks on behalf of the Cinder team for all his past work on the project. cheers, brian

2 1

[ptg][nova][cinder] x-p meeting minutes
by Sylvain Bauza 13 Nov '19

13 Nov '19

Excerpts taken from https://etherpad.openstack.org/p/shanghai-ptg-cinder We discussed two items : 1/ Improving replication - When a failover happens volumes are no longer usable in Nova. - Nova should start re-attaching volumes after a failover? - Why can't we detach and attach the volume? Data that is in flight would be lost. - Question about boot from volume. In that case the instance is dead anyway because access to the volume has been lost. - Could go through the shutdown, detach, attach, reboot path. - Problem is that detach is going to fail. Need to force it or handle the failure. - We aren't sure that Nova will allow a detach of a boot volume. - Also don't currently have a force detach API. - AGREE: Need to figure out how to pass the force to os-brick to detach volume and when rebooting a volume. 2/ Nova bug for images created from encrypted volumes - Nova is not creating a new key for encrypted images but the deletion policy metadata can allow a key to be deleted wehn it is still in use by other images or even volumes - Nova needs to clone the keys when doing create image. - The nova team thinks that we have found a bug that needs to be fixed. We just need to open a bug. - ACTION Cinder to open a bug against Nova. (can you also ping lyarwood when it's open?) -S

3 2

[keystone] Shanghai forum/PTG recap
by Colleen Murphy 13 Nov '19

13 Nov '19

While the keystone team didn't itself meet at last week's PTG, I posted a recap of the event from a keystone perspective here: http://www.gazlene.net/shanghai-forum-ptg.html Hope it's a useful summary for those who couldn't attend in-person. Colleen

1 0

deployment failed
by SHAMBHU KUMAR 13 Nov '19

13 Nov '19

Dear Sir/mam i'm facing the issue while deployment of triple o no valid host found error code 500 Can you please eloborate in this matter because i'm stuck heere Your support will be highly appriciated..

2 1

[scientific-sig] IRC meeting today - Shanghai roundup and Supercomputing 2019
by Stig Telfer 13 Nov '19

13 Nov '19

Greetings all - We have a Scientific SIG meeting in about an hour’s time (2100 UTC) in channel #openstack-meeting. Everyone is welcome. Agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_12th_20… <https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_November_12th_20…> We are going to cover a trip report from Shanghai and planning for the many activities coming up next week in Supercomputing. Cheers, Stig

1 0

[nova][ironic][ptg] Resource tracker scaling issues
by Balázs Gibizer 12 Nov '19

12 Nov '19

* COMPUTE_RESOURCE_SEMAPHORE blocks instance creation on all nodes (on the same host) while the _update_available_resource runs on all nodes. On 3500 baremetal nodes _update_available_resource takes 1.5 hour. * Do we still need _update_available_resource periodic task to run for ironic nodes? * Reduce the scope of the COMPUTE_RESOURCE_SEMAPHORE lock * https://review.opendev.org/#/c/682242/ * https://review.opendev.org/#/c/677790/ * changing a locking scheme is frightening => we need more testing Agreement: * Do a tempest test with a lot of fake ironic node records to have a way to test if changing the locking scheme breaks anything * Log a bug and propose a patch for having a per-node lock instead of the same object for all the ResourceTrackers * See also whether concurrency helps * Propose a spec if you really want to pursue the idea of being somehow inconsistent with data by not having a lock Cheers, gibi

8 11

[tripleo][ptg]
by Wesley Hayutin 12 Nov '19

12 Nov '19

Greetings, Thanks for all those Red Hatters who attended the OpenStack Summit and PTG in Shanghai! A special thanks to those who presented their topics and discussed work items with the folks in attendance. As the current PTL for TripleO I will do my best here to summarize those conversations and items others should be made aware of. Over the course of Thursday and Friday roughly 7-10 folks discussed the identified topics [1], at the following times with my raw notes attached [2]. My apologies if I did not accurately represent your topic here, please feel free to correct me. Thursday, Giulio Fidente: Edge ( SDS storage ) Guilio walked us through some background work with regards to support for storage in remote sites / edge deployments. Working through support for Cinder was straight forward enough with no real collaboration required. Support for ceph copy on write for nova guests was also added with the glance image added to remote sites. Where Guilio needed input was with regards to having change the ctrl plane config for glance for each remote site [3]. This ctrl plane update would force operators to put the cloud in maintenance mode for a stack update. It was determined this could not be avoided at this time. It was noted that the TripleO simplification project and rework puppet-apply, please help us achieve that by reviewing the following two topics [4][5]. Thanks Giulio! Thursday, Guilio Fidente: Virtual IP / Storage Guilio walked us through some challenges with hosting a shared file system on remote/edge sites using manilla. The idea was to use Ganesha translation with CephFS. The proposal was that Ganesha and pacemaker would be managed in the ctrl plane but there was a question with regards to the virtual ip on edge sites. This was an interesting conversation that ended up with a suggestion from Kevin Carter to use a host-only local route on the edge to properly route the ip. This seemed to everyone to be a very clever solution to the problem :) Thanks Guilio, Thanks Kevin! Thursday, Martin Schuppert: Nova CellV2 multicell Martin walked the group through the current status and plans for the multicell implementation. Background: Nova multicells are used to help scale a cloud and partition it in such a way to get the messaging queue closer to the compute cell, essentially rabbit, galera, collector, vnc proxy and a number of compute nodes. This architecture is already in use but with only one default cell, pike was the switch to cellv2. The work started in Stein and continued through train using a similar approach as DCN. Some of the specs are that there is one cell per stack that is initially created from an export of the central stack, more ansible is place for the deployment as well. Two different architectures were noted, all cells in one heat stack [6], and one that splits the cell controllers and computes into different heat stacks w/ multiple stacks on the edge sites [7]. The development work for updates is complete and upgrades is still a WIP. Plans for the future included integrating TLS everywhere and enabling storage in the cell ( cinder, ceph, glance). Tony Breeds pointed out this architecture should just work in multiarch but would like the teams help in designing / advice while creating a test environment. Please review the following patches [13] Thanks Martin!! We tried to get more folks to switch to their topics to Thursday but were not able to. On to Friday. Friday, Edge ( DCN ) roadmap: David Paterson This conversation was informally walked through Thursday mainly with Arkady and Guilio and was followed up on Friday with a joint edge session regarding booting edge nodes. Several questions were raised on Thursday regarding the networking and connectivity for edge sites as it relates to provisioning. Validations were discussed as a way to address the minimum requirements for booting edge nodes. David did not end up presenting here, but was available at the joint session. See the “edge booting” section later in the document for details. Friday, Backup and Restore: Carlos Camacho The project started in Newton. Initially the backup consisted of a database dump and files being backed up for a defined set of use cases. In the field it was discovered that customers had many different kinds of deployments and the feature did not work well for all the intended use cases. An improved plan included to move to full disk image backups utilizing REAR [8]. Carlos also noted that customers are now trying to use ( or misuse ) this feature to perform baremetal to virt migrations. One of the current issues with the current solution include that it’s not clear how services behave after backup and restore.. E.G. OSD mons. Wes Hayutin noted that we have an opportunity to test the full image backup and restore solution by moving to a more image based internal CI system currently being designed by Jesse Pretorious and others. Thanks Carlos!! Friday, Failure Domains: Kevin Carter Unfortunately Kevin was in high demand across PTG events and was unable to present this topic. This should be discussed in a mid-cycle ( virtual or in person ) and written up as a blueprint. Essentially Kevin is proposing in large deployments to allow some number or percentage of nodes to fail a deployment while not failing the entire deployment. If a few non-critical nodes fail a large scale deployment TripleO should be better able to handle that, report back and move on. It was pointed out to me there is a related customer bug <https://bugzilla.redhat.com/show_bug.cgi?id=1721089> as well. Thanks Kevin!! Friday, Cross project: Edge Booting: Julia Kreger You can find notes on this session here [9]. I will only summarize the questions proposed on earlier edge ( DCN ) topic. With regards to when does TripleO need to support redfish there was no immediate or extremely urgent requests ( please correct me if I do not have the correct information there). Redfish IMHO did seem to be a nice improvement when compared to IPMI. This was my first introduction to Redfish, and I of course curious what steps we had to take in order to CI it. Luckily after doc diving I found several helpful links that include steps with setting Redfish up with our own OVB tooling ( hooray \0/ ). Links can be found here [10], and it seems like others have done some hard work to make that possible so thank you!! Thank you Julia!! Friday, Further TripleO Ansible Integration: Wes Hayutin The idea here would be allow the TripleO project to govern how TripleO is deployed with ansible as an operator. The TripleO project would ship ansible modules and roles that directly import python-tripleoclient to support ansible to cli parity [12]. Using or modeling a new repo, perhaps called tripleo-operator-ansible would be used to host these modules and roles and include the same requirements and features of tripleo-ansible’s linting, molecule tests, and auto documentation. This could tie in well with an initiative from Dan Macpherson to ship Ansible playbooks as part of our OSP documentation. Julia Kreger noted that we should not ignore the Ansible OpenStackSDK for part of the deployment process which is a very valid point. Most everyone at the PTG agreed this was a good direction moving forward and would help consolidate the public and internal tooling around TripleO’s CLI in ansible. Thanks Dan, Julia!! Friday, TLS-E standalone gate: Ade Lee Ade Lee walked us through a proposal to help test and CI TLS upstream which has been very difficult to date ( I can personally vouch for this ). Using a two node setup upstream with one node as the IPA server and the other a TripleO standalone deployment. The keystone team is setting the right example for other projects and teams that are finding it difficult to keep outside patches from breaking their code, and that is to find a way to get something voting and gating upstream even if it’s not installed and deployed in the same exact ways customers may use it. Please help by reviewing the keystone / security teams patches here [14] Thanks Ade!! Friday, Octavia tempest plugin support in TripleO-CI: Chandan Kumar Chandan was off fighting battles with the infra team and other projects. Here are some of his notes: Have a rdo third party standalone job with full octavia tempest triggered against octavia patches from stein onwards. FS062 Look into multinode job for queens releases as a third party from queens and rocky. FS038 Have a support of octavia tempest plugin in os_tempest. We certainly should have a conversation offline regarding these topics. I’ll note the TripleO-CI community meeting or the #tripleo meeting on Tuesdays are a good way to continue collaborating here. Thanks Chandan!! Friday, Replace clouds.yaml with an openrc ansible module: Chandan Kumar Open Question: is this module [15] from the openstack-ansible project something we can reuse in TripleO via tripleo-ansible? Friday, Zuul jobs and ansible roles to handle rpm packaging: Chandan Kumar The background and context can be found here https://pagure.io/zuul-distro-jobs - collection of ansible roles to deal with rpm packaging https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/r… -> creating rpms from different projects depends-on in commit message Proposal: Move it to zuul-jobs Making the rpm packaging more generic for centos/fedora/rhel and move it to zuul jobs * Move mock and rpmbuild related roles to zuul jobs repo * Adding a mention of third party zuul jobs to main zuul jobs doc * Build-set_registry setup http server and start the job * Details are here <https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019> Thanks Chandan!! This is indeed a very interesting and powerful proposal. We should definitely continue this conversation with the broader community. Did you make it all the way down here? Well done! I should add an easter egg :) Links: [1] https://etherpad.openstack.org/p/tripleo-ussuri-topics [2] https://etherpad.openstack.org/p/tripleo-ptg-ussuri [3] https://blueprints.launchpad.net/tripleo/+spec/split-controlplane-glance-ca… [4 ] https://review.opendev.org/#/q/topic:disable/paunch+(status:open+OR+status:… [5] https://review.opendev.org/#/q/topic:deconstruct/container-puppet+(status:m… [6] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/feature… [7] https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/feature… https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/feature… [8] https://access.redhat.com/solutions/2115051 [9] https://etherpad.openstack.org/p/PVG-ECG-PTG [10] https://github.com/openstack/sushy https://docs.openstack.org/sushy/latest/contributor/index.html#contributing https://docs.openstack.org/sushy-tools/latest/ https://docs.openstack.org/sushy-tools/latest/user/dynamic-emulator.html#sy… [12] https://hackmd.io/caRlGha7SueZxDRcyq9eGA?both [13] https://review.opendev.org/#/q/topic:cellv2+(status:open+OR+status:merged) [14] https://review.opendev.org/#/q/status:open+project:openstack/tripleo-heat-t… [15] https://opendev.org/openstack/openstack-ansible-openstack_openrc [16] https://etherpad.openstack.org/p/PVG-keystone-forum-policy [17 https://datko.pl/zuul.pdf [18] https://github.com/openstack/tripleo-heat-templates/blob/master/README.rst#… [19] https://github.com/openstack/openstack-virtual-baremetal Thanks all!! Wes Hayutin TripleO-PTL

1 0

[ops] next ops meetup : London 7,8 Jan 2020 - approved!
by Chris Morgan 12 Nov '19

12 Nov '19

https://twitter.com/osopsmeetup/status/1194281816468938752?s=20 -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[all][doc] Patches to add --keep-going to sphinx-build (and patches proposed to many many repositories)
by Akihiro Motoki 12 Nov '19

12 Nov '19

Hi, As you may notice, we see a lot of patches which try to add --keep-going to sphinx-build. [0] I have suggestions and questions. 1) First, when reviewing them, keep the following in your mind. * --keep-going is added even when -W option is not used in the sphinx-build command line. -W is recommended in the PTI [1], so ensure to have -W. * Some of them ignores cases where "python setup.py build_sphinx" is still used. If it is a good chance to clean them up and use "sphinx-build" consistently. 2) Why do we need to remove the build directory for releasenotes? Some of them propose to add "rm -rf releasenotes/build"? (for example [2]) I cannot understand why this needs to be added. Do we really want to call "rm -rf <build dir>"? I know it is needed in some repositories due to various reasons, but generally speaking it make the documentation build longer and they are unnecessary. I tried to get the reason from the authors in reviews but they just say it is simple and it can be added at the same time. Thus, I would like to ask it more broadly in the list. 3) What is the recommended way to get a consensus to make this kind of patches which affects many many repositories? It is not productive to ask questions in individual reviews. Some patches are approved fast and questions are pop-up in other patches. It makes it difficult to discuss the real needs and keep many repositories consistent. I am not against all of these changes, but I would like to see more organized way. Thought? Thanks, Akihiro [0] https://review.opendev.org/#/q/message:keeping [1] https://governance.openstack.org/tc/reference/project-testing-interface.htm… [2] https://review.opendev.org/#/c/690956/2/tox.ini

2 1

[nova][ptg] Resurrecting NUMA topology in Placement
by Sylvain Bauza 12 Nov '19

12 Nov '19

We discussed about a long known story https://review.openstack.org/#/c/552924/ The whole agreement during the PTG was to keep things simple with baby steps : - only supporting a few NUMA queries and defer others as unsupported (still supported by legacy NUMATopologyFilter) - The to-be-resurrected spec would be only focus on VCPU/PCPU/MEMORY_MB resource classes and not handle PCI or GPU devices (ie. level-1 tree, no children under the NUMA RPs) Agreement was also there for saying that functional tests should be enough since PlacementFixture already works perfectly. -S

2 3

[infra] Etherpad problem
by Slawek Kaplonski 12 Nov '19

12 Nov '19

Hi, Just at the end of the ptg sessions, neutron etherpad was got broken somehow. Now when I try to open [1] I see only something like: An error occurred The error was reported with the following id: 'igzOahZ6ruH0eSUAWKaj' Please press and hold Ctrl and press F5 to reload this page, if the problem persists please send this error message to your webmaster: 'ErrorId: igzOahZ6ruH0eSUAWKaj URL: https://etherpad.openstack.org/p/Shanghai-Neutron-Planning UserAgent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0 TypeError: r.dropdowns is undefined in https://etherpad.openstack.org/javascripts/lib/ep_etherpad-lite/static/js/p… at line 18' We can open one of the previous versions which is available at [2] but I don't know how we can fix original etherpad or restore version from [2] to be original etherpad and make it working again. Can someone from infra team check that for us maybe? Thx in advance for any help. [1] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning [2] https://etherpad.openstack.org/p/Shanghai-Neutron-Planning/timeslider#16117 -- Slawek Kaplonski Senior software engineer Red Hat

4 4

[ops] meetups team meeting and meetups venue for Jan 2020
by Chris Morgan 12 Nov '19

12 Nov '19

There wll be an OpenStack Ops Meetups team meeting in just under 45 minutes on #openstack-operators. The agenda is here: https://etherpad.openstack.org/p/ops-meetups-team We'd like to see if we can agree to formally accept the offer to host the next OpenStack Operators Meetup in London next January (see https://etherpad.openstack.org/p/ops-meetup-1st-2020) Please attend the meeting if you have feedback for or against this proposal. So far all feedback has been positive and there are no other offers. Note (full disclosure) that this proposal is from my employer. We'll also go over material from last week's summit in Shanghai. Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[ptg][nova][glance] x-p meeting minutes
by Sylvain Bauza 12 Nov '19

12 Nov '19

We only discussed about nova snapshots to dedicated glance stores. Reference is https://review.opendev.org/#/c/641210/ We had concerns on implementing glance storage location strategies in Nova. A counter-proposal was made to only pass the original image ID when calling Glance for a snapshot so then Glance would then use any store the operator wants (eg. a location strategy like "please store the snapshot to a place close to where the original image is"). The agreement was there for the counter-proposal so the Nova change would only be to pass the original glance image ID to the new glanceclient that would support it. Glance folks will provide a new spec revision. -S

1 0

[ptg][nova][keystone] x-p meeting minutes
by Sylvain Bauza 12 Nov '19

12 Nov '19

We only discussed about the large effort needed for the new policies. https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/poli… https://review.opendev.org/#/q/topic:bp/policy-defaults-refresh+(status:ope… We asked the Keystone team to review the first changes in the series as it's very helpful. We had concerns about completion over the Ussuri cycle since the series can be very large and we want to avoid deprecation messages for some APIs if not all the Nova APIs are touched yet. The agreement was to hold a procedural -2 on the API changes and start reviewing (for both Keystone and Nova folks) until all the changes are there. Maybe a runway could help once all the API changes are uploaded. -S

1 0

[nova][cinder] Volume-backed instance disks and some operations that do not support those yet
by Peter Penchev 12 Nov '19

12 Nov '19

Hi, First of all, thanks to everyone involved for all the work on Nova, Cinder, os-brick, and actually all the rest of OpenStack, too! So, yeah, I guess it is kind of weird that I'm asking this on the list just a couple of days after the PTG where I could have asked in person, but here goes :) There seem to still be some quirks with Nova and volume-backed instance disks; some actions on instances are not allowed, others produce somewhat weird results. From a quick look at the code it seems to me that currently these are: - taking a snapshot of an instance (produces a zero-sized file, no real data backed up) - backing an instance up (refuses outright) - rescuing an instance (refuses outright) ...and maybe there are some that I've missed. So, possibly stupid question here, but what are the project's plans about these - is there an intention to implement them at some point, or are there some very, very hard theoreitcal or practical problems (so something like "guess not for the present"), or is somebody working on something? The main reason that I am asking is that we, StorPool, have a shared-storage Cinder driver, and every now and then a customer comes up and asks about one or more of these actions. Every now and then we come back to the idea of writing a vendor-specific Nova image backend, but, first off, we are not really sure whether we want to do this, and second, we are not really sure whether it will be accepted upstream. A couple of years ago people told us "don't do that" and there was some talk about having an image backend for storage drivers supported by libvirt, but that effort seems to have stalled. Of course, we know that in all software projects, including, but certainly not limited to, the more-or-less volunteer free/libre/open-source projects, there are many tasks and many demands on the developers so that it is only natural that not everything is implemented or adapted at once; things happen, priorities shift, people get redirected, nobody else steps up to continue - it happens. With this in mind, where do things stand right now, should we consider writing an image backend, are there other options or plans? So thanks for reading through my ramblings, I guess, and keep up the great work! Best regards, Peter

4 3

[nova][ptg] Support re-configure deleted_on_termination in server
by Sylvain Bauza 12 Nov '19

12 Nov '19

Spec is https://review.opendev.org/#/c/580336/ Most people seem to think this makes sense but realize there are already other ways to do this (snapshot) and therefore it's not totally necessary. The agreement in the room was to post the code up for the change, as this will help sell people on it if it's trivial enough and document the use case (i.e. are there scenarios where this would make life 10x easier?)

1 0

BGP dynamic routing
by Volodymyr Litovka 12 Nov '19

12 Nov '19

Dear colleagues, "BGP dynamic routing" doc (https://docs.openstack.org/neutron/rocky/admin/config-bgp-dynamic-routing.h…) says only about advertisement of routes: "BGP dynamic routing enables advertisement of self-service (private) network prefixes to physical network devices that support BGP such as routers, thus removing the conventional dependency on static routes." and nothing about receiving of routes from external peers. Whether it is ever possible using Neutron to have fully dynamic routing inside the project, both advertising/receiving (and updating VRs configuration) routes to/from remote peers? Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison

3 5

[tacker] No IRC meeting today
by Dharmendra Kushwaha 12 Nov '19

12 Nov '19

Hello Taker team, As we have PTG in last week, lets skip today's weekly meeting. Thanks & Regards Dharmendra Kushwaha

1 0

How to set password
by zhaowx＠jxresearch.com 12 Nov '19

12 Nov '19

hello: When I use `trovestack build-image`, how to set password for the image? thanks zhaowx(a)jxresearch.com

1 0

[nova][ptg] drop the shadow table concept
by Balázs Gibizer 12 Nov '19

12 Nov '19

CERN reported two issues with archive_deleted_rows CLI: * When one record gets inserted into the shadow_instance_extra but didn't get deleted from instance_extra (I know this is in a single transaction but sometimes it happens), needs manual cleanup on the database * Also there could be two cells running this command at the same time fighting for the API db lock, TODOs: * tssurya to report bugs / improvements on archive_deleted_rows CLI based on CERN's experience with long table locking * mnaser to report a wishlist bug / specless bp about one step db purge CLI which would skip the shadow tables Cheers, gibi

4 6

[octavia][ptg] Summary of Shanghai PTG Discussion
by Adam Harwell 11 Nov '19

11 Nov '19

Fellow Octavians, We covered a lot of ground during this PTG, met a number of new folks, and got a lot of valuable feedback. I'll do my best to summarize here what was discussed. 1. Metrics 1. It would be nice to expose metrics for pools/members, though we would like to get a better understanding of the requirements / use-cases. 2. We should publish metrics using some mechanism (plugin/driver). 1. The default would be "database" and would handle the existing API-exposed metrics. 2. Additional drivers would be loaded in parallel, and might include Monasca/Ceilometer/Prometheus drivers. 3. We will switch our metrics internally to use a delta system instead of absolute values from HAProxy. This will allow us to publish in a more sane way in the future. This would not change the way metrics are exposed in the existing API. 2. Notifications 1. We will need to create a spec and gather community feedback. 2. Initial observation indicates the need for two general paths, which will most likely have their own driver systems: 1. provisioning_status changes (including all create/update/delete events) 2. operating_status changes (member up/down, etc) 3. We would provide the entire object in the notification, similar to what other services do. 4. Most likely the default driver(s) would use oslo.notify. 3. Availability Zone Support (Multi-Zone Fault Tolerance) 1. Make at least a story for tracking this, if it doesn't already exist. 2. Allow a single LB to have amphorae in multiple zones. 3. Existing patch: https://review.opendev.org/#/c/558962/ 4. Availability Zone Support (Compute AZ Awareness) 1. Make at least a story for tracking this, if it doesn't already exist. 2. Allow placing LBs in specific zones: 1. When zones are geographically separated, LBs should exist in the same zone as the members they support 2. When zones are logically separated (like PCI compliance zones, etc), users may need to place them specifically. 3. A new parameter `availability_zone` will be added to the LB create API. It will allow the user to select which Octavia AZ to use. 4. A new API section will be added for creating/configuring/listing Octavia AZs. This will allow a linkage between Compute AZs and Amphora Management Network, along with other possible options in the future. Admins can create/update, and users can list zones. 5. Update clients to support this, including further polluting the `openstack availability zone list` command to include `--loadbalancers` zones. 5. Python 2 EOL 1. Remove all jobs that test Python 2 (or update them if they're not duplicates). 2. Remove six compatibility code, which should simplify string handling significantly. 6. More Flavor Capabilities 1. Image Tag (to allow different amp images per flavor) 2. Availability Zone (to allow compute AZ pinning) 3. Management Network (to go with compute AZ) 4. Metadata (allow passing arbitrary metadata through to compute) 7. TLS Protocol/Cipher API Support 1. Allow users to select specific protocols/ciphers as a whitelist. 2. Stories: 1. Ciphers: https://storyboard.openstack.org/#!/story/2006627 2. Protocols: https://storyboard.openstack.org/#!/story/2006733 8. Performance Tuning 1. HAProxy: There are a number of knobs/dials that can be adjusted to make HAProxy behave more efficiently. Some of these that we could look at more are around TLS options, and around multiprocessing/threading. The latter will probably need to wait for us to switch to HAProxy 2.0. 2. Image Metadata: There are flags that could be added to our amphora image's metadata that might improve performance. To be further researched. 9. Testing 1. Team to evaluate existing non-voting jobs for promotion to voting. 1. Agreement was made with the Barbican team to promote both side's co-gating jobs to voting. 2. Team to evaluate merging or pruning some jobs to reduce the overall set that run on each change. 3. Grenade needs a few changes: 1. Switch to python3. 2. Upgrade to Zuul v3. 3. Test additional operations on existing LBs (old amp image), not just traffic. 4. Test more than just the most recent amphora image against the current control-plane code. Use periodic jobs for this. 5. Fix the Zuul grafana dashboard for Octavia test history. 10. Jobboard 1. This continues to be a priority for Ussuri. 2. Put together a priority list of patches specifically for jobboard. 11. HAProxy 2.0 1. A number of features are gated behind the new version, including multi-process and HTTP/2 support. 2. Need to reach out to distributions to push for backports (to cloudarchive for Ubuntu, and whatever similar thing for CentOS). 3. Possibly add an element to allow building new versions from source. 4. Perform version-based validation of options on the API-side of the amphora driver. 1. Inspect and cache Glance metadata for the LB's amphora image to get version data. 2. Provide the metadata string for the operator from our disk-image-create script. The full etherpad from the PTG including the notes I've summarized here is available at https://etherpad.openstack.org/p/octavia-shanghai-U-ptg if further review is desired. Thanks to everyone who participated, and best of luck on this (hopefully) productive new cycle! --Adam Harwell

1 0

[neutron] Bug Deputy Report Nov. 4-11
by Ryan Tidwell 11 Nov '19

11 Nov '19

Hello neutrinos, here is the bug deputy report for the week of Nov. 4th: High: * https://bugs.launchpad.net/neutron/+bug/1851659 "removing a network from a DHCP agent removes L3 rules even if it shouldn't" This was found on stable/rocky, we should obviously see if it can be reproduced on master, stein, and train as well. There does seem to be a workaround, but VM's lost connectivity for a brief period. Medium: * https://bugs.launchpad.net/neutron/+bug/1851500 ""test_show_port_chain" conflicts with Flow Classifier" This is a gate issue, with what appears to be some instability with a test. Initial triage indicates it's an intermittent issue. * https://bugs.launchpad.net/neutron/+bug/1851194 "FWaaSv2 configures iptables with invalid port name" This similar to https://bugs.launchpad.net/neutron/+bug/1798577, FWaaSv2 seems to be referencing the wrong port names. RFE: * https://bugs.launchpad.net/neutron/+bug/1851609 Add an option for graceful l3 agent shutdown -Ryan Tidwell

1 0

Change Volume Type, but in use
by Sinan Polat 11 Nov '19

11 Nov '19

Hi, I am using Ceph as the backend for Cinder. Within Ceph we have defined 2 RBD pools (ssdvolumes, sasvolumes). In OpenStack I created 2 Volume Types (ssd, sas). Each Volume Type has property "volume_backend_name='tripleo_ceph_<ssd|sas>'". In the Cinder configuration I have the following backends configured: [tripleo_ceph_ssd] backend_host=hostgroup volume_backend_name=tripleo_ceph_ssd volume_driver=cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=openstack rbd_pool=ssdvolumes [tripleo_ceph_sas] backend_host=hostgroup volume_backend_name=tripleo_ceph_sas volume_driver=cinder.volume.drivers.rbd.RBDDriver rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=openstack rbd_pool=sasvolumes As you might have noticed, the backend name (tripleo_ceph_ssd) and the RBD pool name (ssdvolumes, not ssd) does not match. So far, we do not have any problems. But I want to correct the names and I do not want to have the mismatch anymore. So I want to change the value of key volume_backend_name for both Volume Types (tripleo_ceph_ssd => tripleo_ceph_ssdvolumes). I tried the following: $ openstack volume type show 80cb25ff-376a-4483-b4f7-d8c75839e0ce +--------------------+----------------------------------------+ | Field | Value | +--------------------+----------------------------------------+ | access_project_ids | None | | description | | | id | 80cb25ff-376a-4483-b4f7-d8c75839e0ce | | is_public | True | | name | ssd | | properties | volume_backend_name='tripleo_ceph_ssd' | | qos_specs_id | None | +--------------------+----------------------------------------+ $ $ openstack volume type set --property volume_backend_name='tripleo_ceph_ssdvolumes' 80cb25ff-376a-4483-b4f7-d8c75839e0ce Failed to set volume type property: Volume Type is currently in use. (HTTP 400) (Request-ID: req-5efaa5b7-910f-4802-8494-3115cfc4ab93) Command Failed: One or more of the operations failed $ How to solve my problem? Thanks! Sinan

4 6