- openstack-discuss - lists.openstack.org

[all][tc][forum] Summarizing Ussuri cycle community goal discussion in Forum and PTG
by Ghanshyam Mann 19 Nov '19

19 Nov '19

Hello Everyone, We discussed Ussuri goals in Forum as well as in PTG. I would like to summarize the discussions here. * Forum[1]: Three goals were discussed in details: 1. Drop Python 2.7 Support We discussed keeping the CI/CD support for Swift which is the only project keeping the py2 support. Swift needs the devstack to keep installing on py2 env with the rest of the services on py3 (same as old jobs when Swift was on py2 by default on devstack). There is no oslo dependency from swift and all the other dependency will be capped for py2 version. Requirements check job currently checks that if openstack/requirements list two entries for a requirement (one for <=2.7 and one for >) that the repo under test also has both entries. smcginnis already pushed the changes[2] to handle dual python version requirements. Everything else will go as discussed in ML[3] and this is already accepted as Ussuri goal[4]. 2. Project Specific New Contributor & PTL Docs - As per feedback in Forum sessions, this is a good goal which will make documentation more consistent. All the projects should edit their contributor.rst to follow a more complete template and adjust/add PTL documentation. - This is accepted as a pre-approved as Ussuri goal - Kim Hindhart is working on getting EU funding for people to work on OpenStack and they like consistent documentation. - diablo_rojo already updated the goal proposal patch and it is up to get wider feedback. 3. Switch remaining legacy jobs to Zuul v3 and drop legacy support - Grenade job is not yet on zuulv3 which is required to finish first. - Few projects waiting for big projects finishing the zuulv3 migration first. - This needs more work and can be a "pre-approved" thing for V, and this would be split to focus on the Grenade work in U. Other than above 3 goals, there were few more ideas for goal candidate and good to go in goal backlogs etherpad: - cdent: stop using paste, pastedeploy and WSME Note from Chris: This does not need to be a community goal as such but requires the common solution from TC WSME is still used, has contributions, and at least a core or two - cmurphy: Consistent and secure default policies https://etherpad.openstack.org/p/PVG-keystone-forum-policy Going with pop-up team first - support matrix documentation to be consistent across projects. going with pop-up team (fungi can propose the pop-up team in governance) first Richard Pioso (rpioso) to help fungi on this once consistent framework is identified, the pop-up team can expire with the approval of a related cycle goal for implementing it across remaining projects * PTG[5]: We all agreed to select the Ussuri goal asap because the Ussuri cycle already started and projects are waiting for the final goals. TC discussed this further in PTG and agreed on below two goals to be selected as the Ussuri goal. 1. Drop Python 2.7 Support - Already Accepted. 2. Project Specific New Contributor & PTL Docs - Under Review Goal "Switch remaining legacy jobs to Zuul v3 and drop legacy support" will be pre-selected for V cycle, that does not mean to stop the review on proposed goal or any ongoing work. All ongoing efforts will continue on this. [1] https://etherpad.openstack.org/p/PVG-ussuri-goal-forum [2] https://review.opendev.org/#/c/693631/ [3] http://lists.openstack.org/pipermail/openstack-discuss/2019-October/010371.… [4] https://governance.openstack.org/tc/goals/selected/ussuri/drop-py27.html [5] L76 https://etherpad.openstack.org/p/PVG-TC-PTG -gmann

1 0

[magnum] Kubernetes cluster issue
by Namrata Sitlani 19 Nov '19

19 Nov '19

Hello Folks, >From Thursday last week (Nov 14), Magnum is unable to spin up working Kubernetes clusters. We run on Rocky Openstack release. All our Kubernetes pods show CrashLoopBackOff status. We use the following commands to create the Kubernetes clusters : http://paste.openstack.org/show/786348/ . The deployment fails with the following output : http://paste.openstack.org/show/786287/. We tried deploying Kubernetes v1.13.12, v1.14.8, v1.15.5 and v1.16.2 without success. However, if we use version v1.14.6 we can successfully deploy our clusters. Unfortunately, we cannot use v1.14.6 in production because it is not patched for the CVE-2019-11253 vulnerability . Since this stopped working for us on Thursday, we think that the image update https://hub.docker.com/r/openstackmagnum/kubernetes-apiserver/tags done 6 days ago is the culprit. (We previously deployed clusters with versions v1.15.5 and v1.14.8 successfully.) Can you please confirm our findings and help us find a way forward? We can provide more logs if needed. Thank you very much, Namrata Sitlani

2 1

[RabbitMQ][cinder] Listen to messages
by Blom, Merlin, NMU-OI 19 Nov '19

19 Nov '19

Hey there, it seems to me as if ask.openstack.org is down, so I ask my question here: I'd like to listen to oslo messages from cinder as I do for neutron and octavia to know what is going on. For me the following code worked for neutron: EXCHANGE_NAME = os.getenv('EXCHANGE_NAME', 'neutron') ROUTING_KEY = os.getenv('ROUTING_KEY', 'notifications.info') QUEUE_NAME = os.getenv('QUEUE_NAME', 'messaging_queue') BROKER_URI = os.getenv('BROKER_URI', 'UNDEFINED') BROKER_PASSWORD = os.getenv('BROKER_PASSWORD', '') class Messages(ConsumerMixin): def __init__(self, connection): self.connection = connection return def get_consumers(self, consumer, channel): exchange = Exchange(EXCHANGE_NAME, type="topic", durable=False) queue = Queue(QUEUE_NAME, exchange, routing_key=ROUTING_KEY, durable=False, auto_delete=True, no_ack=True) return [consumer(queues=[queue], callbacks=[self.on_message])] def on_message(self, body, message): try: print(message) except Exception as e: log.info(repr(e)) if __name__ == "__main__": log.info("Connecting to broker {}".format(BROKER_URI)) with BrokerConnection(hostname=BROKER_URI, userid='messaging', password=BROKER_PASSWORD, virtual_host='/'+EXCHANGE_NAME, heartbeat=4, failover_strategy='round-robin') as connection: Messaging(connection).run() BrokerConnection.connection.close() But on the cinder vhost (/cinder) I can't find an exchange that the code is working on. (cinder, cinder-backup, .) I tried using the rabbitmq tracer: <https://www.rabbitmq.com/firehose.html> https://www.rabbitmq.com/firehose.html And got all the cinder messages but I don't want to use it in production because of performance issues. Does anyone have an idea how to find the correct exchange for the notification info queue in cinder? Cheers, Merlin

2 2

Trove Image issue
by rabia.shaheen＠xflowresearch.com 18 Nov '19

18 Nov '19

Hi Team, I have deployed kolla-ansible(Stein) with Trove enable and using Trove prebuild images to build the Database VM but VM is constantly stuck in build state. I am not sure how to use prebuild image key which is in https://opendev.org/openstack/trove/src/branch/master/integration/scripts/fi les/keys folder. Can you please guide me regarding the key usage for prebuild images (http://tarballs.openstack.org/trove/images/) To build my own trove image on kolla-ansible, is there any specific guide available for it? Warm Regards, Rabia Shaheen Lead Engineer, xFlow Research Inc. +923075462720 (GMT+5) <mailto:rabia.shaheen@xflowresearch.com> rabia.shaheen(a)xflowresearch.com <http://www.xflowresearch.com/> www.xflowresearch.com

2 1

how to update back-end storage online
by yj x 18 Nov '19

18 Nov '19

Hi, I want to implement a qemu block driver which like qeme/block/iscsi.c. Here are my questions: some targets have be attached to guest system, and then the targets info have changed. How can I update the connection to targets online? I mean the disk in guest system does not change, just update the connection to the back-end storage, and the operation can't affect guest system. I can't find a nova-api or libvirt-api to meet my needs now. Does anyone help me? Thanks.

1 0

[keystone] post-PTG virtual meeting reminder
by Colleen Murphy 18 Nov '19

18 Nov '19

Hi keystoners, As a reminder, we'll be holding our post-PTG meeting tomorrow at 14:00 UTC (with the daylight savings time change that makes it 6:00 PST (RIP me) / 9:00 EST / 19:30 IST). Last week I briefly floated the idea of rescheduling it but quickly decided there was not enough notice, so we're holding it at the day and time that we've had planned for the last few weeks. Our agenda and notes: https://etherpad.openstack.org/p/keystone-shanghai-ptg We'll try jitsi.org again, I think the technical issues we were having last time were on my end and not related to the platform: https://meet.jit.si/keystone-ptg Please please please review the roadmap board and assign yourself to items you have been working on and update their status: https://tree.taiga.io/project/keystone-ussuri-roadmap/kanban . We'll be going over the board together tomorrow, and updating it ahead of time will save us time as a group. Colleen

1 0

[manila] No IRC meeting on Nov 7 and Nov 21 2019
by Goutham Pacha Ravi 18 Nov '19

18 Nov '19

Hello Zorillas and interested stackers, Due to a part of our community attending the Open Infrastructure Summit+PTG (Nov 4-8, 2019) and KubeCon+CloudNativeCon (Nov 18-21, 2019), I propose that we cancel the weekly IRC meetings on Nov 7th and Nov 21st. If you'd like to discuss anything during these weeks, please chime in on freenode/#openstack-manila, or post to this mailing list. Thanks, Goutham

1 1

Shanghai PTG Team Photos
by Kendall Waters 18 Nov '19

18 Nov '19

Hi everyone, Thank you for attending the Project Teams Gathering in Shanghai! If your team took a team picture, you can find a copy of the photo file in this Dropbox folder: https://www.dropbox.com/sh/1my6wdtuc1hf58o/AACU49pjWxzFNzcZJgjLG8n1a?dl=0 If you are unable to open Dropbox, please send me an email with which team photo you are looking for and I can send you the file directly. Cheers, Kendall Kendall Waters OpenStack Marketing & Events kendall(a)openstack.org

1 0

[Openstack] FOSDEM 2020 Virtualization & IaaS Devroom CfP
by Piotr Kliczewski 18 Nov '19

18 Nov '19

Friendly reminder that there are 2 weeks before the submission deadline. Room day update: This year Virt and IaaS room will be on the 2nd of February. See you all at FOSDEM!

1 0

Scheduler sends VM to HV that lacks resources
by Albert Braden 18 Nov '19

18 Nov '19

If I create 20 VMs at once, at least one of them fails with "Exceeded maximum number of retries." When I look at the logs I see that the scheduler sent the VM to a host that doesn't have enough CPU "Free vcpu 14.00 VCPU < requested 16 VCPU." https://paste.fedoraproject.org/paste/6N3wcDzlbNQgj6hRApHiDQ I thought that this must be caused by a race condition, so I stopped the scheduler and conductor on 2 controllers, and then created 20 more VMs. Now I see the logs only on controller 3, and some of the failures are now saying "Unable to establish connection to <LB>" but I still see the single scheduler sending VMs to a host that lacks resources "Free vcpu 14.00 VCPU < requested 16 VCPU." https://paste.fedoraproject.org/paste/lGlVpfbB9C19mMzrWQcHCQ I'm looking at my nova.conf but don't see anything misconfigured. My filters are pretty standard: enabled_filters = RetryFilter,AvailabilityZoneFilter,CoreFilter,RamFilter,ComputeFilter,ComputeCapabilitiesFilter,ImagePropertiesFilter,ServerGroupAntiAffinityFilter,ServerGroupAffinityFilter,DifferentHostFilter,SameHostFilter What should I be looking for here? Why would a single scheduler send a VM to a host that is too full? We have lots of compute hosts that are not full: https://paste.fedoraproject.org/paste/6SX9pQ4V1KnWfQkVnfoHOw This is the command line I used: openstack server create --flavor s1.16cx120g --image QSC-P-CentOS6.6-19P1-v4 --network vg-network --max 20 alberttestB

5 14

[neutron][drivers] Drivers meeting cancel
by Slawek Kaplonski 18 Nov '19

18 Nov '19

Hi, I can't attend next drivers meeting on Friday, 22.11.2019. I know also that 2 other members of drivers team will not be able to be on the meeting so as we will not have quorum on this meeting, lets cancel it. See You on the meeting next week, on 29.11.2019 where (I hope) we should have quorum even if it's just after Thanksgiving. -- Slawek Kaplonski Senior software engineer Red Hat

1 0

[cloudkitty] 18/11 IRC meeting recap
by Luka Peschke 18 Nov '19

18 Nov '19

Hello, This is the recap for today's IRC meeting of the cloudkitty team. The agenda can be found at [1] and the logs can be found at [2]. CloudKitty 11.0.1 ================= CloudKitty 11.0.1 has been released. It includes a fix for security issue on the GET /v1/dataframes and GET /v2/dataframes endpoints which had been introduced during the train development cycle. Various updates =============== Two patches had been discussed during the previous meeting: one for developer documentation for scope fetchers, and one allowing to group results by timestamps on GET /v2/summary. Both have been merged. API improvements ================ Two patches improving the API are currently under review: - The first one updates the way oslo.context and oslo.policy are used. External reviews on this one would be very helpful [3] - The second one improves the way various drivers are loaded in the v2 API. It is available at [4] Standalone cloudkitty dashboard =============================== Julien Pinchelimouroux (julien-pinchelim) has been working on a standalone dashboard (which will also support keystone authentication)for cloudkitty. A 0.1.0 release should happen during in Q4 2019. Some screenshots are available at [5] Cheers, -- Luka Peschke (peschk_l) [1] https://etherpad.openstack.org/p/cloudkitty-meeting-topics [2] http://eavesdrop.openstack.org/meetings/cloudkitty/2019/cloudkitty.2019-11-… [3] https://review.opendev.org/#/c/692333/ [4] https://review.opendev.org/#/c/686393/ [5] https://kutt.it/khA6yF

1 0

[horizon] Changing the release model to cycle-with-intermediary
by Akihiro Motoki 18 Nov '19

18 Nov '19

Hi, I just proposed a patch to change the horizon release model to cycle-with-intermediary. https://review.opendev.org/#/c/694772/ It was discussed during the Shanghai PTG. Horizon provides GUI to users. On the other hand, it is a library for horizon plugins. When horizon plugins would like to use recent changes or to avoid bugs, they need to consume beta releases of horizon. More frequent releases of horizon would make more sense, so I am proposing the release model change. If there are concerns, reply to this thread or drop comments in the review mentioned above. Thanks, Akihiro Motoki (irc: amotoki)

1 0

[neutron] Removing neutron-interconnection out of stadium
by Slawek Kaplonski 16 Nov '19

16 Nov '19

Hi neutrinos, As we discussed during the Shanghai PTG I just proposed changes to move neutron-interconnection project out of stadium to "x/" namespace. So it will not be official neutron project anymore after those changes. Patches for that are in [1] and [2] I also proposed to remove neutron-interconnection api-ref from neutron-lib. Patch is here [3]. Please review it :) [1] https://review.opendev.org/#/c/694478/ [2] https://review.opendev.org/#/c/694480/ [3] https://review.opendev.org/#/c/694466/ -- Slawek Kaplonski Senior software engineer Red Hat

4 5

[oslo][i18n][pbr] get_available_languages() always only returns ['en_US']
by info＠dantalion.nl 16 Nov '19

16 Nov '19

Hello, Across several projects I have noticed that both with unit tests or while the service is running calling olso_i18n.get_available_languages() only returns ['en_US'] (which is inserted as default). Even though the projects I have tested this on have several languages available in the locale directory. Calling any of the python setup.py extract_messages / compile_catalog / update_catalog / install commands does not solve this. However, when I manually copy the .mo files into /usr/share/locale/**/LC_MESSAGES it works as expected. My questions are: Surely there must be a less manual method to properly install all locale files but what is it? Why aren't the locale files installed by pbr when python setup.py install is called? I hope anyone knows the answers to these questions. Kind regards, Corne Lukken (Dantali0n)

1 0

[nova] Displayed state of VM when nova-compute is down/unresponsive.
by Laurent Dumont 15 Nov '19

15 Nov '19

Hey everyone, We had a discussion with some colleagues at work. There was some confusion over the expected behavior of Openstack/Nova regarding the state of VMs on a compute that is down(or one where Nova is in a "bad" state and unable to update properly). Right now, it seems that the VMs will stay in the last state they we're seen. We we're wondering if there was a way to expose the fact that the underlying hypervisor is down? Something like a "Warning : no data from compute since xx:xx:xx" I did not see any documentation regarding a possible configuration option somewhere but a lot of posts with people with similar questions. I understand that the state of the VM shouldn't be changed based on the status of a compute - but exposing the fact that the state itself is not current might be good middle-ground. I do see a possible issue with the fact that the hypervisor itself is not known if the User/Project is not admin. Is anyone aware of anything similar in the past? Thanks!

2 1

Intel 3rd Party CI - Offline until Monday due to upgrades.
by Gresham, Timothy 15 Nov '19

15 Nov '19

Infrastructure upgrades are occurring in the lab which hosts Intel's OpenStack 3rd party CI. These upgrades will require us to take our CI offline for the weekend. Jobs covering the following areas will be offline. * Persistent memory * Tap as a Service * NFV * PCI * SRIOV Jobs covering Cinder/RSD should not be impacted. Service is expected to be restored Monday afternoon Pacific time. We will send out another email once service has been restored. Tim Gresham Cloud Engineer - Intel Corporation Intel Architecture, Graphics, and Software

1 0

[Neutron] OVS forwarding issues
by Volodymyr Litovka 15 Nov '19

15 Nov '19

Dear colleagues, just faced an issue with Openvswitch, which looks strange for me. The problem is that any particular VM receives a lot of packets, which are unicasted: - from other VMs which reside on the same host (let's name them "local VMs") - to other VMs which reside on other hosts (let's name them "remote VMs") Long output from "ovs-ofctl dump-flows br-int" which, as far as I can narrow, ends there: # ovs-ofctl dump-flows br-int |grep " table=94," |egrep "n_packets=[123456789]" cookie=0xaf6b1435fe826bdf, duration=2952350.695s, table=94, n_packets=291494723, n_bytes=40582103074, idle_age=0, hard_age=65534, priority=1 actions=NORMAL coming to normal processing (classic MAC learning). Looking into br-int MAC-table (ovs-appctl fdb/show br-int) shows, that there are really no MAC addresses of remote VMs and br-int behaves in the right way, flooding unknown unicast to all ports in this L2 segment. Of course, there is br-tun which connected over vxlan to all other hosts and to br-int: Bridge br-tun Controller "tcp:127.0.0.1:6633" is_connected: true fail_mode: secure Port "vxlan-0a960008" Interface "vxlan-0a960008" type: vxlan options: {df_default="true", in_key=flow, local_ip="10.150.0.5", out_key=flow, remote_ip="10.150.0.8"} [ ... ] Port br-tun Interface br-tun type: internal Port patch-int Interface patch-int type: patch options: {peer=patch-tun} but MAC table on br-tun is empty as well: # ovs-appctl fdb/show br-tun port VLAN MAC Age # Finally, packets get to destination, while being copied to all ports on source host, which is serious security issue. I do not think so conceived by design, I rather think we missed something in configuration. Can anybody point me where we're wrong and help with this issue? We're using Openstack Rocky and OVS 2.10.0 under Ubuntu 16.04. Network configuration is: @controller: # cat /etc/neutron/plugins/ml2/ml2_conf.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [ml2] type_drivers = flat,vxlan tenant_network_types = vxlan mechanism_drivers = l2population,openvswitch extension_drivers = port_security,qos,dns_domain_ports [ml2_type_flat] flat_networks = provider [ml2_type_geneve] [ml2_type_gre] [ml2_type_vlan] [ml2_type_vxlan] vni_ranges = 400:400000 [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true @agent: # cat /etc/neutron/plugins/ml2/openvswitch_agent.ini |egrep -v "^$|^#" [DEFAULT] verbose = true [agent] tunnel_types = vxlan l2_population = true arp_responder = true extensions = qos [ovs] local_ip = 10.150.0.5 bridge_mappings = provider:br-ex [securitygroup] firewall_driver = openvswitch enable_security_group = true enable_ipset = true [xenapi] Thank you. -- Volodymyr Litovka "Vision without Execution is Hallucination." -- Thomas Edison

3 4

[kolla] Shutdown ordering of MariaDB containers?
by Eddie Yen 15 Nov '19

15 Nov '19

Hi everyone, I want to ask about the order of shutdown MariaDB (or mean controller) node. For previous steps we found is usually shutdown slaves first, then master [1]. But we found that the MariaDB still get container restarting issue even I followed the step after booting up the cluster. Below is that I did when shutdown/boot up controller. 1. Shutdown the slaves first, then master 2. Boot master first, then slaves. For looking which one is master, we usually looking for the haproxy log and find which mariadb node that the last session access the DB. Or looking for which mariadb container has "--wsrep-new-cluster" in BOOTSTRAP_ARGS. Does anyone has experience about this? Many thanks, Eddie. [1] https://bugs.launchpad.net/kolla-ansible/+bug/1712087

2 2

[oslo] Shanghai Wrapup
by Ben Nemec 15 Nov '19

15 Nov '19

I wrote up a bunch of thoughts about Oslo stuff in Shanghai: http://blog.nemebean.com/content/oslo-shanghai Hopefully I covered everything (and accurately) but if I messed anything up I blame jet lag. :-P -Ben

2 1

Re: [nova][ptg] Flavor explosion
by Brin Zhang(张百林) 14 Nov '19

14 Nov '19

Hi all, Based on the discussion on the Train PTG, and reference to the records on the etherpad and ML, I was updated that SPEC, and I think there are some details need to be discussed, and I have listed some details, if there are any other things that I have not considered, or if some place that I thoughtless, please post a discussion. List some details as follows, and you can review that spec in https://review.opendev.org/#/c/663563. Listed details: - Don't change the model of the flavor in nova code and in the db. - No change for operators who choose not to request the flavor extra specs group. - Requested more than one flavor extra specs groups, if there are different values for the same spec will be raised a 409. - Flavor in request body of server create that has the same spec in the request ``flavor_extra_specs_group``, it will be raised a 409. - When resize an instance, you need to compare the ``flavor_extra_specs_group`` with the spec request spec, otherwise raise a 400. ---------------------------------------------------------------------------------------- Items: [lists.openstack.org代发][nova][ptg] Flavor explosion original spec: https://review.opendev.org/#/c/663563 with -2s The first round of discussion was resulted in no agreement. Then on Friday we revisited the issue based on mdbooth's proposal about composability. Agreement in the room: * Do not try to change the model of the flavor in nova code and in the db. * Define a "ComposableFlavorBit" (bikeshed on the name please) REST API entity that can hold any kind of flavor bits (extra specs, normal flavor fields), propose some format in the spec for it. This entity can only be created by the admin by default * Extend the server create REST API to allow the end user to specify what "ComposableFlavorBit"s she wants to add to the "base" flavor she used in the create request. * The nova api then merges the "ComposableFalvorBit"s with the base flavor and embed the resulted flavor object into the instance. * Do a similar thing for resize TODOs: * brinzhang (with possible help from yawang) to re-write the spec Cheers, gibi

2 2

[nova] Thoughts on exposing exception type to non-admins in instance action event
by Matt Riedemann 14 Nov '19

14 Nov '19

tl;dr: What do people think about storing and showing the *type* of exception that is recorded with a failed instance action event (like a fault) to the owner of the server who may not be an admin? Details: As noted here [1] and recreated here [2] the instance action event details that a non-admin owner of a server sees do not contain any useful information about what caused the failure of the action. Here is an example of a failed resize from that paste (this is what the non-admin owner of the server would see): $ openstack --os-compute-api-version 2.51 server event show vm2 req-11487504-da59-411b-b3b8-267bebe9b0d2 -f json -c events { "events": [ { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "cold_migrate", "result": "Error" }, { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "conductor_migrate_server", "result": "Error" } ] } Super useful, right? In this case scheduling failed for the resize so the instance is not in ERROR status which means the user cannot see a fault message with the NoValidHost error either. The admin can see the traceback in the failed action event list: $ openstack --os-compute-api-version 2.51 server event show 3ef043ea-e2d7-4565-a401-5c758e149f23 req-11487504-da59-411b-b3b8-267bebe9b0d2 -f json -c events { "events": [ { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "traceback": " File \"/opt/stack/nova/nova/conductor/manager.py\", line 301, in migrate_server\n host_list)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 367, in _cold_migrate\n raise exception.NoValidHost(reason=msg)\n", "event": "cold_migrate", "result": "Error" }, { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "traceback": " File \"/opt/stack/nova/nova/compute/utils.py\", line 1411, in decorated_function\n return function(self, context, *args, **kwargs)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 301, in migrate_server\n host_list)\n File \"/opt/stack/nova/nova/conductor/manager.py\", line 367, in _cold_migrate\n raise exception.NoValidHost(reason=msg)\n", "event": "conductor_migrate_server", "result": "Error" } ] } So when the admin gets the support ticket they can at least tell that scheduling failed and then dig into why. My idea is to store the exception *type* with the action event, similar to the recorded instance fault message for non-NovaExceptions [3] which will show to the non-admin owner of the server if the server status is ERROR or DELETED [4]. We should record the exc_val to get a prettier message like "No valid host was found." but that could leak details in the error message that we don't want non-admins to see [5]. With what I'm thinking, the non-admin owner of the server could see something like this for a failed event: { "finish_time": "2019-11-13T16:18:27.000000", "start_time": "2019-11-13T16:18:26.000000", "event": "cold_migrate", "result": "Error", "details": "NoValidHost" } That's pretty simple, doesn't leak details, and at least indicates to the user that maybe they can retry the resize with another flavor or something. It's just an example. This would require a microversion so before writing a spec I wanted to get general feelings about this in the mailing list. I accept that it might not really be worth the effort so that's good feedback if it's how you feel (I'll only cry a little). [1] https://review.opendev.org/#/c/693937/2/nova/objects/instance_action.py [2] http://paste.openstack.org/show/786054/ [3] https://github.com/openstack/nova/blob/20.0.0/nova/compute/utils.py#L101 [4] https://github.com/openstack/nova/blob/20.0.0/nova/api/openstack/compute/vi… [5] https://bugs.launchpad.net/nova/+bug/1851587 -- Thanks, Matt

5 11

Order filters by cost
by Albert Braden 14 Nov '19

14 Nov '19

I'm working on a list of filters ordered by cost. This is what I have so far. Does this look reasonably correct for Rocky? Cheap Filters: AllHostsFilter - does no filtering. It passes all the available hosts. AvailabilityZoneFilter - filters hosts by availability zone. AggregateInstanceExtraSpecsFilter - checks aggregate metadata set with aggregate_instance_extra_specs. All hosts are passed if no extra_specs are specified. AggregateCoreFilter - filters hosts by CPU core number with per-aggregate cpu_allocation_ratio setting. AggregateRamFilter - filters hosts by RAM with per-aggregate ram_allocation_ratio setting. AggregateDiskFilter - filters hosts by disk allocation with per-aggregate disk_allocation_ratio setting. AggregateNumInstancesFilter - filters hosts by number of instances with per-aggregate max_instances_per_host setting. AggregateIoOpsFilter - filters hosts by I/O operations with per-aggregate max_io_ops_per_host setting. AggregateMultiTenancyIsolation - isolate tenants in specific aggregates. AggregateTypeAffinityFilter - limits instance_type by aggregate. AggregateImagePropertiesIsolation - isolates hosts based on image properties and aggregate metadata. DifferentHostFilter - allows the instance on a different host from a set of instances. SameHostFilter - puts the instance on the same host as another instance in a set of instances. ComputeFilter - passes all hosts that are operational and enabled. NumInstancesFilter - filters compute nodes by number of running instances. IoOpsFilter - filters hosts by concurrent I/O operations. More Expensive Filters: ServerGroupAntiAffinityFilter - This filter implements anti-affinity for a server group. ServerGroupAffinityFilter - This filter works the same way as ServerGroupAntiAffinityFilter. The difference is that when you create the server group, you should specify a policy of 'affinity'. ImagePropertiesFilter - filters hosts based on properties defined on the instance's image. Doc on setting image properties is here: https://docs.openstack.org/glance/rocky/admin/useful-image-properties.html IsolatedHostsFilter - filter based on isolated_images, isolated_hosts and restrict_isolated_hosts_to_isolated_images flags. SimpleCIDRAffinityFilter - allows a new instance on a host within the same IP block. MetricsFilter - filters hosts based on metrics weight_setting. Most Expensive Filters: PciPassthroughFilter - Filter that schedules instances on a host if the host has devices to meet the device requests in the 'extra_specs' for the flavor. ComputeCapabilitiesFilter - checks that the capabilities provided by the host compute service satisfy any extra specifications associated with the instance type. NUMATopologyFilter - filters hosts based on the NUMA topology requested by the instance, if any. JsonFilter - allows simple JSON-based grammar for selecting hosts. Deprecated Filters: Please don't use any of these; they are obsolete in Rocky: RetryFilter - filters hosts that have already been attempted for scheduling. Obsolete since Queens. RamFilter - filters hosts by their RAM. Obsolete since Pike. CoreFilter - filters based on CPU core utilization. Obsolete since Pike. DiskFilter - filters hosts by their disk allocation. Obsolete since Pike.

2 1

[all] requirements-check failures
by Sean McGinnis 14 Nov '19

14 Nov '19

Hey everyone, You may have noticed some odd failures with the requirements-check job on your patches lately. The requirements team is aware of this issue and are working to get it resolved ASAP. I believe things should be good again once https://review.opendev.org/694248 lands. So for the time being, please hold off on doing rechecks on these patches. This job should only be running for patches that touch any of the requirements files. As a workaround for now, if your patch can make a change without modifying requirements, that should bypass the need to run this job. Another alternative would be to add: Depends-on: https://review.opendev.org/694248 to your commit message, but hopefully that will not be necessary once this patch makes it through and the test is fixed. Sorry for the inconvenience this has caused. Sean

2 2

[nova][ptg] Resource provider delete at service delete
by Balázs Gibizer 14 Nov '19

14 Nov '19

ML thread: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007135.html Agreements in the room: * Check ongoing migration and reject the delete if migration with this compute having the source node exists. Let operator confirm the migrations * Cascade delete providers and allocations in placement. * in case of evacuated instances this is the right thing to do * in any other dangling allocation case nova has the final thrut so nova has the authority to delete them. * Document possible ways to reconcile Placement with Nova using heal_allocations and eventually the audit command once it's merged. Cheers, gibi

2 4