June 2021 - openstack-discuss - lists.openstack.org

[ patrole ] Understanding RBAC Permission Exceptions
by darumaseye 12 Jul '21

12 Jul '21

Hello, I am a researcher studying security of the Openstack's Policies. I'm very interested in the Patrole project and I would like to ask you some questions. In the documentation it is stated that Patrole compares two different types of results: - an expected one ( derived from oslo.policy ) - an actual one ( derived from an actual request to the API ). Given that, Patrole's tests can return 3 different values: 1) "Success" if actual result and expected result are both True or both False; 2) "RbacOverPermissionException" if actual result is True but expected result is False; 3) "RbacUnderPermissionException" in the other case. I can't understand in which cases the tests can return a value different from "Success". As far as I know, an API call should always be validated internally by oslo.policy's rules, before being allowed. So, in order for an API call to be accepted, oslo.policy's rules must allow that API call. It seems to me that the "expected" result ( derived from oslo.policy ) is always included in the actual result. Hoping that everything I said is correct, I would like to ask you: What issue is allowing such a strange behavior in Openstack APIs ? Why the expected results can be different from the actual ones? Are there publicly available examples showing "Failure" values? In general, are there any publicly available test cases that i can study to understand this? I Would like to thank you very much in advance; Best Regards, Jacopo

3 3

[all][infra][qa] Retiring Logstash, Elasticsearch, subunit2sql, and Health
by Clark Boylan 09 Jul '21

09 Jul '21

Hello everyone, Xenial has recently reached the end of its life. Our logstash+kibana+elasticsearch and subunit2sql+health data crunching services all run on Xenial. Even without the distro platform EOL concerns these services are growing old and haven't received the care they need to keep running reliably. Additionally these services represent a large portion of our resource consumption: * 6 x 16 vcpu + 60GB RAM + 1TB disk Elasticsearch servers * 20 x 4 vcpu + 4GB RAM logstash-worker servers * 1 x 2 vcpu + 2GB RAM logstash/kibana central server * 2 x 8 vcpu + 8GB RAM subunit-worker servers * 64GB RAM + 500GB disk subunit2sql trove db server * 1 x 4 vcpu + 4GB RAM health server To put things in perspective, they account for more than a quarter of our control plane servers, occupying over a third of our block storage and in excess of half the total memory footprint. The OpenDev/OpenStack Infra team(s) don't seem to have the time available currently to do the major lifting required to bring these services up to date. I would like to propose that we simply turn them off. All of these services operate off of public data that will not be going away (specifically job log content). If others are interested in taking this on they can hook into this data and run their own processing pipelines. I am sure not everyone will be happy with this proposal. I get it. I came up with the idea for the elasticsearch job log processing way back at the San Diego summit. I spent many many many hours since working to get it up and running and to keep it running. But pragmatism means that my efforts and the team's efforts are better spent elsewhere. I am happy to hear feedback on this. Thank you for your time. Clark

10 13

[Octavia dashboard] Impossible to fill the subnet field when creating a load balancer
by Taltavull Jean-Francois 09 Jul '21

09 Jul '21

Hi All, On my OSA Victoria deployment (on Ubuntu 20.04), the "Create Load Balancer" form does not allow me to fill the mandatory "subnet" field and therefore I can't create a load balancer with Horizon. On the other hand, Octavia works fine and I can create load balancers with the CLI, Terraform, etc. Has one of you already faced such a situation ? Best regards, Jean-François

3 6

[Ironic] Vendor-neutral Disk names
by Mahnoor Asghar 06 Jul '21

06 Jul '21

Dear all, There is [a proposal][1] in the metal3-io/baremetal-operator repository to extend the hardware RAID configuration to support ‘physical_disks’ and ‘controller’ fields in the 'target_raid_config' section. The user should be able to specify the disks for RAID configuration, in a vendor-agnostic way. (This requirement comes from the Airship project.) The names of the disks should be indicative of the physical location of the disks within the server. An algorithm to construct disk names is therefore needed, for this purpose. One possible algorithm was found in the [inventory module][2] of the [Redfish Tacklebox scripts][3]. To construct a disk name, it uses Redfish properties, specifically the ‘Drive’ resource> ‘Physical Location’ object> ‘Part Location’ property> ‘Service Label’ property. ([Link][4] to code) ([Link][5] to Redfish ‘Drive’ resource) If this property is empty, the resource type (String uptil the first dot encountered in the @odata.type field), and the ‘Id’ properties of the Drive resource are used to construct the disk name. ([Link][6] to code) For example, if the 'Drive'.'Physical Location'.'Part Location'.'Service Label' field is ‘Disk.Bay1.Slot0’, this is what the disk name will be. If this field is empty, and the resource name is ‘Drive’ and the resource ‘Id’ is ‘Disk.Bay1.Slot0’, the disk name will be: ‘Drive: Disk.Bay1.Slot0’. We would like to understand the values different vendors give in: - ‘Drive’ resource> ‘Physical Location’ object> ‘Part Location’ property> ‘Service Label’ property - The resource type for a Drive (@odata.type field) - The ‘Id’ property of the Drive resource Also, it would be helpful to understand the existing logic used by vendors to construct the disk names, including the (Redfish or other) properties used. This is so that a consensus can be reached for an algorithm to construct disk names. Any suggestions are welcome, thank you so much! [1]: https://github.com/metal3-io/metal3-docs/pull/148 [2]: https://github.com/DMTF/Redfish-Tacklebox/blob/master/redfish_utilities/inv… [3]: https://github.com/DMTF/Redfish-Tacklebox [4]: https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce4… [5]: https://redfish.dmtf.org/schemas/v1/Drive.v1_12_1.json [6]: https://github.com/DMTF/Redfish-Tacklebox/blob/e429f70a79cfe288756618498ce4… Best regards, Mahnoor Asghar Software Design Engineer xFlow Research Inc. mahnoor.asghar(a)xflowresearch.com www.xflowresearch.com

4 6

[wallaby][nova] CPU topology and NUMA Nodes
by Ammad Syed 06 Jul '21

06 Jul '21

Hi, I am using openstack wallaby on ubuntu 20.04 and kvm. I am working to make optimized flavor properties that should provide optimal performance. I was reviewing the document below. https://docs.openstack.org/nova/wallaby/admin/cpu-topologies.html I have two socket AMD compute node. The workload running on nodes are mixed workload. My question is should I use default nova CPU topology and NUMA node that nova deploys instance by default OR should I use hw:cpu_sockets='2' and hw:numa_nodes='2'. Which one from above provide best instance performance ? or any other tuning should I do ? The note in the URL (CPU topology sesion) suggests that I should stay with default options that nova provides. Currently it also works with libvirt/QEMU driver but we don’t recommend it in production use cases. This is because vCPUs are actually running in one thread on host in qemu TCG (Tiny Code Generator), which is the backend for libvirt/QEMU driver. Work to enable full multi-threading support for TCG (a.k.a. MTTCG) is on going in QEMU community. Please see this MTTCG project <http://wiki.qemu.org/Features/tcg-multithread> page for detail. Ammad

5 9

[nova] spec review day
by Balazs Gibizer 05 Jul '21

05 Jul '21

Hi, During the today's meeting it came up that we should have a spec review day around M1 (and later one more before M2). As M1 is 27th of May, I propose to do a spec review day on 25th of May, which is a Tuesday. Let me know if the timing does not good for you. The rules are the usual. Let's use this day to focus on open specs, trying to reach agreement on as many thing as possible with close cooperation during the day. Cheers, gibi

1 3

[Kolla-Ansible] Regarding stability of Ironic deployment with Kolla-Ansible
by Amit Mahajan 01 Jul '21

01 Jul '21

Hi All, We are exploring Kolla-Ansible for our multinode production grade deployment. In our setup, some workloads will also run on bare-metal, so Ironic is the key requirement. We came across the following text on the Kolla-Ansible documentation page for Rocky release. "Ironic works well in Kolla, though it is not currently tested as part of Kolla CI, so may be subject to instability." Our query is that is deployment of Ironic using Kolla-Ansible production grade for the latest OpenStack release?

3 6

[kayobe][victoria] no module named docker - deploy fail after deploy successful
by Tony Pearce 01 Jul '21

01 Jul '21

I had a successful deployment of Openstack Victoria via Kayobe with an all-in-one node running controller and compute roles. I wanted to then add 2 controller nodes to make 3 controllers and one compute. The 2 additional controllers have a different interface naming so I needed to modify the inventory. I checked this ansible documentation to figure out the changes I’d need to make [1]. The first try, I misunderstood the layout because kayobe tried to configure the new nodes interfaces with incorrect naming. In my first try I tried this inventory layout: Move existing configuration: [kayobe config] / Inventory / * To [kayobe config] / Inventory / Ciscohosts / Create new from copy of existing:[kayobe config] / Inventory / Ciscohosts / > [kayobe config] / Inventory / Otherhosts / The Otherhosts had its own host file with the 2 controllers and group_vars/controllers network interface configuration as per these two hosts. But anyway it didnt achieve the desired result so I rechecked the ansible doc [1] and decided to do this another way as follows: In my second try I first reversed the inventory change: Delete the new dir: [kayobe config] / Inventory / Otherhosts / Move back the config: [kayobe config] / Inventory / Ciscohosts / * > [kayobe config] / Inventory / Delete the empty dir: [kayobe config] / Inventory / Ciscohosts / Then create host_vars for the two individual hosts: [kayobe config] / Inventory / host_vars / cnode2 [kayobe config] / Inventory / host_vars / cnode3 And updated the single hosts inventory file [kayobe config] / Inventory / hosts This seemed to work fine, the “kayobe overcloud host configure” was successful and the hosts interfaces were set up as I desired. The issue came when doing the “kayobe overcloud service deploy” and failed with "/usr/bin/python3", "-c", "import docker” = ModuleNotFoundError: No module named 'docker' for all three nodes, where previously it (the deployment) had been successful for the all-in-one node. I do not know if this task had run or skipped before but the task is run against "baremetal" group and controllers and compute are in this group so I assume that it had been ran successfully in previous deployments and this is the weird thing because no other changes have been made apart from as described here. Verbose error output: [3] After the above error, I reverted the inventory back to the “working” state, which is basically to update the inventory hosts and removed the 2 controllers. As well as remove the whole host_vars directory. After doing this however, the same error is still seen /usr/bin/python3", "-c", "import docker” = ModuleNotFoundError. I logged into the host and tried to run this manually on the CLI and I see the same output. What I don’t understand is why this error is occurring now after previous successful deployments. To try and resolve /workaround this issue I have tried to no avail: - recreating virtual environments on all-in-one node - recreating virtual environments on ACH - deleting the [kolla config] directory - deleting .ansible and /tmp/ caches - turning off pipelining After doing the above I needed to do the control host bootstrap and host configure before service deploy however the same error persisted and I could not work around it with any of the above steps being performed. As a test, I decided to turn off this task in the playbook [4] and the yml file runs as follows: [2]. This results in a (maybe pseudo) successful deployment again, in a sense that it deploys without failure because that task does not run. After this was successful in deploying once again as it had previously had been, I added the two controller nodes using the “host_vars” and then I was able to successfully deploy again with HA controllers. Well, it is successful apart from Designate issue due to Designate already having the config [5]. I can log in to the horizon dashboard and under system information I can see all three controllers there. Could I ask the community for help with: 1. Regarding the kayobe inventory, is anything wrong with the 2nd attempt in line with Kayobe? 2. Has anyone come across this docker issue (or similar within this context of failing after being successful) and can suggest? I repeatedly get these odd issues where successful deployments then fail in the future. This often occurs after making a config change and then rolling back but the roll back does not return to a working deployment state. The fix/workaround for me in these cases is to “kayobe overcloud service destroy --yes-i-really-really-mean-it” and also re-deploy the host. [1] Best Practices — Ansible Documentation <https://docs.ansible.com/ansible/2.8/user_guide/playbooks_best_practices.ht…> [2] modified Checking docker SDK version# command: "{{ ansible_python.execut - Pastebin.com <https://pastebin.com/xXJFHc0N> [3] TASK [prechecks : Checking docker SDK version] ********************************* - Pastebin.com <https://pastebin.com/ZmLs4LUZ> [4] /home/cv-user/kayobe-victoria/venvs/kolla-ansible/share/kolla-ansible/ansible/roles/prechecks/tasks/package_checks.yml [5] TASK [designate : Update DNS pools] ******************************************** - Pastebin.com <https://pastebin.com/USiCAXpT> Kind regards,

2 2

[nova][placement] Openstack only building one VM per machine in cluster, then runs out of resources
by Jeffrey Mazzone 01 Jul '21

01 Jul '21

Hello, I am installing Openstack Ussuri and am running into an issue when using Availability Zones. I initially thought it was a quota issue but that no longer seems to be the case. I started a thread on serverfault and was recommended to submit these questions here as well. Here is the original link: https://serverfault.com/questions/1064579/openstack-only-building-one-vm-pe… The issue is still, I can successfully build vms on every host, but only one vm per host. The size of the initial vm does not matter. Since I posted the thread above, I have redeployed the entire cluster, by hand, using the docs on openstack.org<http://openstack.org>. Everything worked as it should, I created 3 test aggregates, 3 test availability zones, with no issues for about a month. All of a sudden, the system reverted to no longer allowing more than one machine to be placed per host. There has been no changes to the controller. I have enabled placement logging now so I can see more information but I don’t understand why its happening. Example. Start with a host that has no vms on it: ~# openstack resource provider usage show 3f9d0deb-936c-474a-bdee-d3df049f073d +----------------+-------+ | resource_class | usage | +----------------+-------+ | VCPU | 0 | | MEMORY_MB | 0 | | DISK_GB | 0 | +----------------+-------+ Create 1 vm with 4 cores ~# openstack resource provider usage show 3f9d0deb-936c-474a-bdee-d3df049f073d +----------------+-------+ | resource_class | usage | +----------------+-------+ | VCPU | 4 | | MEMORY_MB | 0 | | DISK_GB | 0 | +----------------+-------+ The inventory list for that provider is: ~# openstack resource provider inventory list 3f9d0deb-936c-474a-bdee-d3df049f073d +----------------+------------------+----------+----------+----------+-----------+--------+ | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | +----------------+------------------+----------+----------+----------+-----------+--------+ | VCPU | 16.0 | 1 | 64 | 0 | 1 | 64 | | MEMORY_MB | 1.5 | 1 | 515655 | 512 | 1 | 515655 | | DISK_GB | 1.0 | 1 | 7096 | 0 | 1 | 7096 | +----------------+------------------+----------+----------+----------+-----------+--------+ Trying to start another vm on that host fails with the following log entries: scheduler.log "status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider conductor.log Failed to schedule instances: nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. placement.log Over capacity for VCPU on resource provider 3f9d0deb-936c-474a-bdee-d3df049f073d. Needed: 4, Used: 8206, Capacity: 1024.0 As you can see, the used value is suddenly 8206 after a single 4 core vm is placed on it. I don’t understand what im missing or could be doing wrong. Im really unsure where this value is being calculated from. All the entries in the database and via openstack commands show the correct values except in this log entry. Has anyone experienced the same or similar behavior? I would appreciate any insight as to what the issue could be. Thanks in advance! -Jeff M

5 14

Fwd: [TRIPLEO] - ZUN Support in TripleO
by Lokendra Rathour 01 Jul '21

01 Jul '21

Hello Everyone, We are curious in understanding the usage of ZUN Sevice in TripleO with respect to which we have questions as below: 1. Does TripleO Support ZUN? 2. If not then, is there any alternative to deploy containerize services in TripleO? Any support with respect to the questions raised will definitely help us in deciding the tripleO usage. -- ~ Lokendra skype: lokendrarathour

6 7