June 2022 - openstack-discuss - lists.openstack.org

[doc] question/enhancement-request on Sphinx theme used by openstack docs
by Waines, Greg 10 Jun '22

10 Jun '22

Hey ... I am on the technical steering committee for the openstack starlingx team and also provide primary technical review for our docs at https://docs.starlingx.io/ . docs.starlingx.io basically uses the openstack docs team's Sphinx theme for building our html docs ... see https://opendev.org/starlingx/docs . docs.starlingx.io is a very large doc suite. We are interested in improvements to the Table-of-Contents (TOC) navigation and presentation ... but we have limited Sphinx expertise. Specifically we are looking for the TOC support to have - more subsection header levels in TOC // such that user has full view of where he currently is in the doc hierarchy - highlighting to indicate currently selected section - separate scroll bar for TOC // ... so you don't lose sight of selected TOC selection when you scroll down on the main page which has large content. Have you ever considered such doc improvements? Did you have Sphinx expertise to do such improvements ? let us know, Greg.

2 1

[nova] local LVM volume on compute hosts
by Paul Harrison 10 Jun '22

10 Jun '22

I have installed a small test cluster using openstack-ansible (xena) and would like to use local LVM volumes for storing instance volumes - and following https://docs.openstack.org/nova/latest/configuration/config.html and (e.g. https://cloudnull.io/2017/12/nova-lvm-an-iop-love-story/) I have the libvirt section of my /etc/nova/nova.conf as below. -8<--- [libvirt] inject_partition = -2 inject_password = False inject_key = False virt_type = kvm live_migration_with_native_tls = true live_migration_scheme = tls live_migration_inbound_addr = 172.29.236.5 hw_disk_discard = ignore disk_cachemodes = none images_type = lvm images_volume_group = nova volume_clear = zero -8<—— however, the nova service continues to use cinder volumes from my storage host to store instances. Is there something else that needs to be configured to make this work, or is it obsolete?

2 7

[magnum] Docker Swarm errors on Yoga Ubuntu using Install Guide cut-n-paste
by Vince Mulhollon 10 Jun '22

10 Jun '22

I'm trying to use Magnum on a fresh Yoga install on Ubuntu to run some non-load balanced docker swarm workloads. Its a long story but the parts of the swarm workload are heterogeneous so a cluster would just get in the way. Everything else about the cluster is working fine, all is well, etc. I'm a small time VMware and OS admin with years of experience but absolute zero experience with Magnum. Also I have "some" experience with Docker Swarm clusters mostly created and assembled with Ansible. So I know quite well how to look at log files or configs in a general sense, but I have no idea what to expect from Magnum specifically as a "glue" technology. I've followed every detail of the "Magnum Installation Guide" for Yoga as seen at: https://docs.openstack.org/magnum/yoga/install/index.html When provisioning a docker swarm, I very literally cut and paste from: https://docs.openstack.org/magnum/yoga/install/launch-instance.html#provisi… cluster creation fails every time with "Property allowed_cidrs not assigned" exactly as seen on this mailing list post from 2021-August-20 about ten months ago on a Wallaby install, they were also using Ubuntu. http://lists.openstack.org/pipermail/openstack-discuss/2021-August/024273.h… Advice at that point ranged from, try installing Octavia, which I was trying to avoid and I definitely do NOT want a load balancer in front of that particular workload, to Docker swarm is no longer tested for Magnum and it's in practice a K8S exclusive product. I played around for an afternoon and got nowhere WRT configuring the allowed cidrs for a k8s load balancer on a config that does not use a load balancer and does not use k8s. Heat works fine, its super fun, and one alternative is I could give up on Magnum and use some Heat templates, which I know how to do and seems simpler than getting Magnum to work at this point. Or I could brave the scary looking mostly-undocumented world of kuryr and zun, which I was attempting to avoid by using Magnum which is at least simple and easy to install. Or just keep spinning up manual images and letting ansible turn them into docker swarms for me. But, ideally, I'd like to use Magnum to spin up some Docker Swarms for personal curiosity reasons, if nothing else, it would be very convenient if it worked. So the point of this long ramble is, can anyone successfully running Docker Swarm on Yoga on Ubuntu using a very typical "Install Guide" installation provide a pointer or link or github or something that Google searches is not finding to get it to work? Also I hate to piggyback multiple questions but this paragraph also interests me, does anyone have any information and experience with this: https://docs.openstack.org/magnum/yoga/user/index.html#heat-stack-templates "Sample cluster driver To help developers in creating new COE drivers, a minimal cluster driver is provided as an example. The ‘docker’ cluster driver will simply deploy a single VM running Ubuntu with the latest Docker version installed. It is not a true cluster, but the simplicity will help to illustrate the key concepts. To be filled in" That "To be filled in" part would also be very interesting to me as worst case I could shove the Docker Swarm's workload onto a single Docker host, although then I'm back to orchestrating with Heat and presumably Ansible but via a different path.

1 0

[kolla-ansible][xena] Cell database HA/redundancy?
by Norrie, Andrew 10 Jun '22

10 Jun '22

Hi, We are currently planning some large OpenStack deployments utilizing kolla-ansible and I'm curious what folks are doing with the cell database HA/redundancy. With kolla-ansible (xena) it appears that a loadbalancer setup is only allowed for the default database shard (shard 0)... reference: https://docs.openstack.org/kolla-ansible/latest/reference/databases/mariadb… If we are setting up separate cell database shards with Galera, I'm wondering if there is a convenient work-around or configuration for implementation of HA for these cell databases. In the inventory group_vars directory you can specify the group variables (for each cell database) as: nova_cell_database_address: nova_cell_database_group: but these aren't virtual IP hosts/addresses (non default db shard). This works perfectly fine with a single server cell database but not Galera. If that db server goes down the cell instance information is lost. Many thanks ... Andrew ________________________________ "This e-mail and any accompanying attachments are confidential. The information is intended solely for the use of the individual to whom it is addressed. Any review, disclosure, copying, distribution, or use of the email by others is strictly prohibited. If you are not the intended recipient, you must not review, disclose, copy, distribute or use this e-mail; please delete it from your system and notify the sender immediately."

3 3

Error creating octavia amphora
by Russell Stather 10 Jun '22

10 Jun '22

Hi I am seeing the below error when creating a load balancer. The security group does exist, and it is the correct group (tagged with charm-octavia) What am I missing here to resolve this? 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server octavia.common.exceptions.ComputeBuildException: Failed to build compute instance due to: {'code': 500, 'created': '2022-06-09T10:09:59Z', 'message': 'Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.', 'details': 'Traceback (most recent call last):\n File "/usr/lib/python3/dist-packages/nova/conductor/manager.py", line 654, in build_instances\n scheduler_utils.populate_retry(\n File "/usr/lib/python3/dist-packages/nova/scheduler/utils.py", line 989, in populate_retry\n raise exception.MaxRetriesExceeded(reason=msg)\nnova.exception.MaxRetriesExceeded: Exceeded maximum number of retries. Exceeded max scheduling attempts 3 for instance 524ae27b-1542-4c2d-9118-138d9e7f3770. Last exception: Security group 0b683c75-d900-4e45-acb2-8bc321580666 not found.\n'} 2022-06-09 10:10:03.789 678859 ERROR oslo_messaging.rpc.server

4 6

[ops] details for Ops Meetup, Berlin, Tomorrow, Friday 2022-6-10
by Chris Morgan 09 Jun '22

09 Jun '22

A brief twitter thread here https://twitter.com/osopsmeetup/status/1535004027032940544?s=20&t=AgLzV6Qvg… has some reminders about tomorrow's event. Looking forward to seeing everyone. Chris -- Chris Morgan <mihalis68(a)gmail.com>

1 0

[CLOUDKITTY][OpenInfra summit 2022][June 9th] Presentation on Ceilometer, Gnocchi, and CloudKitty
by Rafael Weingärtner 09 Jun '22

09 Jun '22

Hello CloudKitters! It was a great opportunity to be able to present this talk for you all guys. It was a pleasure to see that so many people are interested in CloudKitty, Ceilometer, and Gnocchi. I really hope that we can all collaborate together to keep improving CloudKitty, and other related systems. Attached is the presentation we used during the presentation, so you all can consult this material. Also, as always, if you need some guidance, let us know. Furthermore, as I mentioned in the presentation, I will provide a step-by-step process on how to create a new metric using Ceilometer Dynamic pollster that can collect/generate/derive (however you want to call it) IOPs information for volumes. The new metric name is going to be called "dynamic_pollster.volume.services.volume.iops". Also, the idea is that we collect some attributes, and we use the attribute called "bootable", which indicates if the volume is bootable or not. Then, if the volume is bootable, we will charge for its IOPS; otherwise, we do not charge it. As I mentioned, this is just a very simple use case. It is not targeting anyone's need in production, but it can be used to get you going quite easily with the pipeline. Requirements: - Ceilometer up and running. Version 15.0.0 (or higher is recommended); - Gnocchi up and running. Version 4.3.0 (or higher). If you want to run this in production, this patch might be very important ( https://github.com/gnocchixyz/gnocchi/pull/1059); it has not been merged yet; - CloudKitty up and running. Version 10.0.0 (or higher). With all systems up and running, you will need to do the following: - Ceilometer - Add the new metric collection in the "polling.yaml" file. You can add it in any of the sections you already have there. Or, you can create its own, as follows: ``` - name: volume_iops_metric interval: 600 meters: - dynamic_pollster.volume.services.volume.iops ``` - After declaring the metrics there, you can create the pollster in the "pollster.d" folder in Ceilometer. The name of the file can be anything, as long as it ends with ".yaml"; for instance, "pollsters.d/volume-iops.yaml". Then, you will need to add the following content. Remember to replace the contents of "<>" with the values of your environment. ``` - name: "dynamic_pollster.volume.services.volume.iops" response_entries_key: "volumes" next_sample_url_attribute: "volumes_links | filter(lambda v: v.get('rel') == 'next', value) | list(value) | value[0] | value.get('href')" sample_type: "gauge" unit: "GB" value_attribute: "volume_type" endpoint_type: "volume" url_path: "/v3/<SERVICE_UUID_FOR_CEILOMETER_USER>/volumes/detail?all_tenants=true&limit=10" headers: "Openstack-API-Version": "latest" project_id_attribute: "os-vol-tenant-attr:tenant_id" metadata_fields: - "status" - "name" - "volume_type" - "availability_zone" - "user_id" - "bootable | str(value)" - "attachments | value or [ { 'server_id': '' } ] | value[0] | value['server_id']" metadata_mapping: "availability_zone": "dynamic_availability_zone" "name": "dynamic_display_name" "volume_type": "dynamic_volume_type" "attachments | value or [{ 'server_id': '' }] | value[0] | value['server_id']": "dynamic_instance_id" "bootable | str(value)": "bootable" value_mapping: "<VOLUME_TYPE_UUID_OR_VOLUME_TYPE_NAME>": "<THE IOPS THAT THIS VOLUME TYPE REPRESENTS>" ``` - After creating that, you will need to customize the file "gnocchi_resources.yaml", which is the file that maps metrics to resources, and attributes that are pushed to the backend. You can also create your own "gnocchi_resources.yaml" file, and then use it in the pipeline configuration in the "pipeline.yaml" file, in the configuration of "publishers". Then, you would have something like "gnocchi://?resources_definition_file=%2Fetc%2Fceilometer%2Fcustom_gnocchi_resources.yaml", assuming the file is called "custom_gnocchi_resources.yaml". In this file, you will need to add the new metric and attribute to be pushed to the metrics backend. The following is an example of "volume" resource declaration in the "gnocchi_resource.yaml" file with the new metric and attribute that we want to capture. ``` - resource_type: volume metrics: <MANY OTHER METRICS HERE> dynamic_pollster.volume.services.volume.iops: attributes: display_name: resource_metadata.(dynamic_display_name|notification_display_name) bootable: resource_metadata.bootable <MANY OTHER ENTRIES HERE> ``` - After the above configuration, you will need to create the new attribute in the Gnocchi resource type. To achieve that, you can issue the following command. ``` gnocchi resource-type update -a bootable:string:false volume ``` - Then, you will need to restart the Ceilometer. - Now, you can issue "gnocchi resource show <VOLUME ID>", and you should see the new metric in the resource, and the new attribute as well. With Ceilometer and Gnocchi configured for the new metric, we can move on to CloudKitty. - The new metric needs to be declared in the "metrics.yml" file. The content should be something similar to the following (adapt as you need). ``` dynamic_pollster.volume.services.volume.iops: unit: IOPS alt_name: disk-volume-iops-usage-hours groupby: - id - user_id - project_id - volume_type - display_name - revision_start - instance_id - availability_zone - bootable metadata: [] extra_args: aggregation_method: max resource_type: volume use_all_resource_revisions: false ``` - Restart CloudKitty processor - With the new metric introduced in CloudKitty processor, you will need to create the rating configuration. In our case, we wanted to rate the volumes marked as bootable for IOPs. The volumes that are not bootable would not be rated. The following are the steps one needs to execute to create the hashmap rating rules. - Create the service ``` openstack rating hashmap service create disk-volume-iops-usage-hours ``` - Then, create the field ``` openstack rating hashmap field create <SERVICE_ID> <FIELD NAME WE WANT TO RATE, IN OUR CASE, IT IS bootable> ``` - After that, we can create the rating rule ``` openstack rating hashmap mapping create --field <FIELD UUID THAT WE CREATE> --value 'true' 1 --type flat ``` - Having done that, you should be able to rate all volumes by IOPS if they have the flag "bootable" set to true. Otherwise, they will not be rated. That is all that one needs to execute to replicate the example I showed during the presentation. If you guys have any doubts or difficulties when using CloudKitty, do not hesitate to ping me. -- Rafael Weingärtner

1 0

[rbac][nova][cinder] Canned roles for service users / inter-service communication (e.g. event submission)
by Christian Rohmann 09 Jun '22

09 Jun '22

Hey openstack-discuss, I posted to the ML quite a while ago about an issue of resized (Cinder) volumes not being propagated to the (Nova) instance. See http://lists.openstack.org/pipermail/openstack-discuss/2021-February/020476…. The issue there was Cinder being not allowed to send the "volume-extended" event (or any event for that matter via the Nova API just using the user token. For this a configurable additional "privileged user" was added to the config quite a while back with https://opendev.org/openstack/cinder/commit/04003d7c513ed4dd5129cbd5ad1af14…. While the functionality then works I suppose there should be canned and maintained RBAC roles for such kind of inter service to service communications, e.g. to emit events to others. Otherwise deployments likely will use admin privileged users ignoring the least privilege principle and creating an large attack surface. And there are quite few of those relations even with the most commonly used services. Cinder -> nova, nova-> cincer, nova->ironic, .... nova-> neutron, .... Are such canned RBAC rules for "special" inter service users on the backlog somewhere? Or am I totally misconceiving the issue here? I know there is https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… and also the question for feedback at https://etherpad.opendev.org/p/rbac-operator-feedback, but that all seems to focus on the impact of roles used by humans / users and not about service roles at all. Regards Christian

2 3

[all][operator][policy] Operator feedback on 'Consistent and Secure RBAC" (new design for RBAC)
by Ghanshyam Mann 09 Jun '22

09 Jun '22

Hello Everyone, As you might know, we are redesigning the OpenStack default RBAC. The new design target two things: 1. 'new defaults (reader role)' 2. "Scope" concept It is hard to explain the details in email but the below doc is a good place to start understanding this: - https://governance.openstack.org/tc/goals/selected/consistent-and-secure-rb… We as a community think 1st target (reader role) is a good thing to do and it will definitely be useful in many cases. But we need feedback on the "Scope" concept. To understand what it is and how it can impact your existing use case/deployment, please ref the documentation mentioned in the etherpad[1] (if there is any question about its design/usage we are planning, feel free to reply here or contact us in #openstack-tc IRC channel). * If you are an operator, we really need your feedback if the 'Scope' concept is a useful thing for your deployment/use-case or not. * If you are attending events have operators also attending (for example, project operator feedback (like nova[2]), forum sessions in berlin summit, ops meetup or any local operator event), please communicate about the required feedback. * Due to various reasons, many of us involved in RBAC work are not travelling to Berlin and we have this topic to be discussed in Berlin ops meetup[3] but we require someone knowing RBAC new design moderate this topic. Please reach out to us if you would like to help. Central Etherpad to collect feedback (this can be used to collect from various forums/places): * https://etherpad.opendev.org/p/rbac-operator-feedback [1] https://etherpad.opendev.org/p/rbac-operator-feedback [2] https://etherpad.opendev.org/p/nova-berlin-meet-and-greet [3]https://etherpad.opendev.org/p/ops-meetup-berlin-2022-planning#L74 -gmann

4 12

[tripleo] gate blocker - master/wallaby - "SELinux boolean os_enable_vtpm does not exist."
by Ronelle Landy 09 Jun '22

09 Jun '22

Hello All, We have a consistent failure on multiple jobs running on check/gate/integration lines on master and wallaby changes - which shows up as a deployment error: FATAL | Enable os_enable_vtpm SELinux boolean for vTPM | standalone | error={"changed": false, "msg": "SELinux boolean os_enable_vtpm does not exist."} Details of the failure are in: https://bugs.launchpad.net/tripleo/+bug/1977873 We are investigating a fix/workaround. Please hold rechecks - we will post out again with updates. Thank you!

1 3