July 2021 - openstack-discuss - lists.openstack.org

[release][tc][neutron] how to EOL unofficial branches under offical repositories
by Akihiro Motoki 01 Jul '21

01 Jul '21

Hi, I have a question about how to mark stable branches as EOL when such branches were not part of official releases at the moment of corresponding releases but the repositories which contain them became official after the release. One example happens in a repository under the neutron team. networking-odl is now part of the neutron governance but as of Ocata release it was not official. Ocata branches under the neutron stadium were marked as EOL recently, but networking-odl was not marked as EOL as it was not an official repository as of Ocata. The openstack/releases repository handles official branches under a specific release, but all branches in official repositories are under control of the openstack/releases repo. I wonder how we can mark stable/ocata branch of networking-odl repo as EOL and delete stable/ocata branch. What is the recommended way to mark such branches described above as EOL? Thanks in advance, Akihiro Motoki (irc: amotoki)

3 3

[Kolla-Ansible] Regarding stability of Ironic deployment with Kolla-Ansible
by Amit Mahajan 01 Jul '21

01 Jul '21

Hi All, We are exploring Kolla-Ansible for our multinode production grade deployment. In our setup, some workloads will also run on bare-metal, so Ironic is the key requirement. We came across the following text on the Kolla-Ansible documentation page for Rocky release. "Ironic works well in Kolla, though it is not currently tested as part of Kolla CI, so may be subject to instability." Our query is that is deployment of Ironic using Kolla-Ansible production grade for the latest OpenStack release?

3 6

[kayobe][victoria] no module named docker - deploy fail after deploy successful
by Tony Pearce 01 Jul '21

01 Jul '21

I had a successful deployment of Openstack Victoria via Kayobe with an all-in-one node running controller and compute roles. I wanted to then add 2 controller nodes to make 3 controllers and one compute. The 2 additional controllers have a different interface naming so I needed to modify the inventory. I checked this ansible documentation to figure out the changes I’d need to make [1]. The first try, I misunderstood the layout because kayobe tried to configure the new nodes interfaces with incorrect naming. In my first try I tried this inventory layout: Move existing configuration: [kayobe config] / Inventory / * To [kayobe config] / Inventory / Ciscohosts / Create new from copy of existing:[kayobe config] / Inventory / Ciscohosts / > [kayobe config] / Inventory / Otherhosts / The Otherhosts had its own host file with the 2 controllers and group_vars/controllers network interface configuration as per these two hosts. But anyway it didnt achieve the desired result so I rechecked the ansible doc [1] and decided to do this another way as follows: In my second try I first reversed the inventory change: Delete the new dir: [kayobe config] / Inventory / Otherhosts / Move back the config: [kayobe config] / Inventory / Ciscohosts / * > [kayobe config] / Inventory / Delete the empty dir: [kayobe config] / Inventory / Ciscohosts / Then create host_vars for the two individual hosts: [kayobe config] / Inventory / host_vars / cnode2 [kayobe config] / Inventory / host_vars / cnode3 And updated the single hosts inventory file [kayobe config] / Inventory / hosts This seemed to work fine, the “kayobe overcloud host configure” was successful and the hosts interfaces were set up as I desired. The issue came when doing the “kayobe overcloud service deploy” and failed with "/usr/bin/python3", "-c", "import docker” = ModuleNotFoundError: No module named 'docker' for all three nodes, where previously it (the deployment) had been successful for the all-in-one node. I do not know if this task had run or skipped before but the task is run against "baremetal" group and controllers and compute are in this group so I assume that it had been ran successfully in previous deployments and this is the weird thing because no other changes have been made apart from as described here. Verbose error output: [3] After the above error, I reverted the inventory back to the “working” state, which is basically to update the inventory hosts and removed the 2 controllers. As well as remove the whole host_vars directory. After doing this however, the same error is still seen /usr/bin/python3", "-c", "import docker” = ModuleNotFoundError. I logged into the host and tried to run this manually on the CLI and I see the same output. What I don’t understand is why this error is occurring now after previous successful deployments. To try and resolve /workaround this issue I have tried to no avail: - recreating virtual environments on all-in-one node - recreating virtual environments on ACH - deleting the [kolla config] directory - deleting .ansible and /tmp/ caches - turning off pipelining After doing the above I needed to do the control host bootstrap and host configure before service deploy however the same error persisted and I could not work around it with any of the above steps being performed. As a test, I decided to turn off this task in the playbook [4] and the yml file runs as follows: [2]. This results in a (maybe pseudo) successful deployment again, in a sense that it deploys without failure because that task does not run. After this was successful in deploying once again as it had previously had been, I added the two controller nodes using the “host_vars” and then I was able to successfully deploy again with HA controllers. Well, it is successful apart from Designate issue due to Designate already having the config [5]. I can log in to the horizon dashboard and under system information I can see all three controllers there. Could I ask the community for help with: 1. Regarding the kayobe inventory, is anything wrong with the 2nd attempt in line with Kayobe? 2. Has anyone come across this docker issue (or similar within this context of failing after being successful) and can suggest? I repeatedly get these odd issues where successful deployments then fail in the future. This often occurs after making a config change and then rolling back but the roll back does not return to a working deployment state. The fix/workaround for me in these cases is to “kayobe overcloud service destroy --yes-i-really-really-mean-it” and also re-deploy the host. [1] Best Practices — Ansible Documentation <https://docs.ansible.com/ansible/2.8/user_guide/playbooks_best_practices.ht…> [2] modified Checking docker SDK version# command: "{{ ansible_python.execut - Pastebin.com <https://pastebin.com/xXJFHc0N> [3] TASK [prechecks : Checking docker SDK version] ********************************* - Pastebin.com <https://pastebin.com/ZmLs4LUZ> [4] /home/cv-user/kayobe-victoria/venvs/kolla-ansible/share/kolla-ansible/ansible/roles/prechecks/tasks/package_checks.yml [5] TASK [designate : Update DNS pools] ******************************************** - Pastebin.com <https://pastebin.com/USiCAXpT> Kind regards,

2 2

[kayobe][kolla-ansible][victoria]
by Tony Pearce 01 Jul '21

01 Jul '21

How do I configure the glance container to use a host-mounted NFS share? It is not covered in the kolla-ansible documentation [1]. What it does say is to set the file path using "glance_file_datadir_volume". I believe that this is relative to the container (not the host). Do I need to mount the local host (nfs) directory as a docker volume for glance as additional step? If so, how do I achieve this with kayobe / kolla-ansible? Some background: After a successful deployment of 3 controller nodes I have one glance service which is dependent on one of these controller nodes being available always. My intent is to provide a NFS share for glance that removes the dependency. [1] OpenStack Docs: Glance - Image service <https://docs.openstack.org/kolla-ansible/victoria/reference/shared-services…> Kind regards, Tony Pearce

2 2

[nova][placement] Openstack only building one VM per machine in cluster, then runs out of resources
by Jeffrey Mazzone 01 Jul '21

01 Jul '21

Hello, I am installing Openstack Ussuri and am running into an issue when using Availability Zones. I initially thought it was a quota issue but that no longer seems to be the case. I started a thread on serverfault and was recommended to submit these questions here as well. Here is the original link: https://serverfault.com/questions/1064579/openstack-only-building-one-vm-pe… The issue is still, I can successfully build vms on every host, but only one vm per host. The size of the initial vm does not matter. Since I posted the thread above, I have redeployed the entire cluster, by hand, using the docs on openstack.org<http://openstack.org>. Everything worked as it should, I created 3 test aggregates, 3 test availability zones, with no issues for about a month. All of a sudden, the system reverted to no longer allowing more than one machine to be placed per host. There has been no changes to the controller. I have enabled placement logging now so I can see more information but I don’t understand why its happening. Example. Start with a host that has no vms on it: ~# openstack resource provider usage show 3f9d0deb-936c-474a-bdee-d3df049f073d +----------------+-------+ | resource_class | usage | +----------------+-------+ | VCPU | 0 | | MEMORY_MB | 0 | | DISK_GB | 0 | +----------------+-------+ Create 1 vm with 4 cores ~# openstack resource provider usage show 3f9d0deb-936c-474a-bdee-d3df049f073d +----------------+-------+ | resource_class | usage | +----------------+-------+ | VCPU | 4 | | MEMORY_MB | 0 | | DISK_GB | 0 | +----------------+-------+ The inventory list for that provider is: ~# openstack resource provider inventory list 3f9d0deb-936c-474a-bdee-d3df049f073d +----------------+------------------+----------+----------+----------+-----------+--------+ | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | +----------------+------------------+----------+----------+----------+-----------+--------+ | VCPU | 16.0 | 1 | 64 | 0 | 1 | 64 | | MEMORY_MB | 1.5 | 1 | 515655 | 512 | 1 | 515655 | | DISK_GB | 1.0 | 1 | 7096 | 0 | 1 | 7096 | +----------------+------------------+----------+----------+----------+-----------+--------+ Trying to start another vm on that host fails with the following log entries: scheduler.log "status": 409, "title": "Conflict", "detail": "There was a conflict when trying to complete your request.\n\n Unable to allocate inventory: Unable to create allocation for 'VCPU' on resource provider conductor.log Failed to schedule instances: nova.exception_Remote.NoValidHost_Remote: No valid host was found. There are not enough hosts available. placement.log Over capacity for VCPU on resource provider 3f9d0deb-936c-474a-bdee-d3df049f073d. Needed: 4, Used: 8206, Capacity: 1024.0 As you can see, the used value is suddenly 8206 after a single 4 core vm is placed on it. I don’t understand what im missing or could be doing wrong. Im really unsure where this value is being calculated from. All the entries in the database and via openstack commands show the correct values except in this log entry. Has anyone experienced the same or similar behavior? I would appreciate any insight as to what the issue could be. Thanks in advance! -Jeff M

5 14

[neutron] Drivers meeting agenda for 2.07.2021
by Slawek Kaplonski 01 Jul '21

01 Jul '21

Hi, We have 2 RFEs to discuss on tomorrow's meeting: https://bugs.launchpad.net/neutron/+bug/1933517 - [RFE][OVN] Create an intermediate OVS bridge between VM and intergration bridge to improve the live-migration process https://bugs.launchpad.net/neutron/+bug/1930866 - [RFE] preventing from deleting a port used by an instance (locked instance can be rendered broken by deleting port) We also have 2 not yet triaged RFEs: https://bugs.launchpad.net/neutron/+bug/1932154 https://bugs.launchpad.net/neutron/+bug/1933222 Please check them, ask questions/comments/whatever You need to triage them so we can discuss on next drivers meetings :) -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

Fwd: [TRIPLEO] - ZUN Support in TripleO
by Lokendra Rathour 01 Jul '21

01 Jul '21

Hello Everyone, We are curious in understanding the usage of ZUN Sevice in TripleO with respect to which we have questions as below: 1. Does TripleO Support ZUN? 2. If not then, is there any alternative to deploy containerize services in TripleO? Any support with respect to the questions raised will definitely help us in deciding the tripleO usage. -- ~ Lokendra skype: lokendrarathour

6 7

[openstack] [DC-DC Setup] [Replication] Correct approach to a DC-DC setup for Openstack (victoria)
by Swogat Pradhan 01 Jul '21

01 Jul '21

Hi, I am trying to setup a DC-DC setup in openstack victoria, I have 2 numbers of all in one setup (controller, compute) with shared mysql and rabbitmq cluster and am using ceph image replication and configuring cinder replication on top of it. So when the 1st node goes down then i will perform a 'cinder failover-host node1' and then do a nova-evacuate to failover to the 2nd DC setup and once the node 1 comes up i will use cinder failback and live migration from node2 to node1. **i am facing some minor issues in this setup which I will ask once I know this is the right approach to the DC-DC concept. Can you please shed some light on this being the right approach or not and if not then how can i improve it? With regards Swogat pradhan

1 1

[all][tc] Technical Committee next weekly meeting on July 1st at 1500 UTC
by Ghanshyam Mann 01 Jul '21

01 Jul '21

Hello Everyone, Technical Committee's next weekly meeting is scheduled for July 1st at 1500 UTC. If you would like to add topics for discussion, please add them to the below wiki page by Wednesday, June 30th, at 2100 UTC. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting -gmann

1 1

[Swift] Object replication failures on newly upgraded servers
by Mark Kirkwood 01 Jul '21

01 Jul '21

HI, I'm in the process of upgrading a Swift cluster from 2.7/Mitaka to 2.23/Train. While in general it seems to be going well, I'm noticing non-zero object replication failures on the upgraded nodes only, e.g: $ curl http://localhost:6000/recon/replication/object {"replication_last": 1622156911.019487, "replication_stats": {"rsync": 40580, "success": 4141229, "attempted": 2081856, "remove": 4083, "suffix_count": 14960481, "failure": 26550, "hashmatch": 4127197, "failure_nodes": {"10.11.18.67": {"obj08": 2348, "obj09": 60, "obj10": 3030, "obj02": 34, "obj03": 25, "obj01": 44, "obj06": 1498, "obj07": 28, "obj04": 69, "obj05": 36}, "10.11.18.68": {"obj03": 6901, "obj01": 293, "obj06": 1901, "obj04": 10281, "obj10": 1}, "10.12.18.76": {"obj10": 1}}, "suffix_sync": 1785, "suffix_hash": 2778}, "object_replication_last": 1622156911.019487, "replication_time": 1094.7836411476135, "object_replication_time": 1094.7836411476135} Examining the logs (/var/log/swift/object.log and /var/log/syslog) these are not throwing up any red flags (i.e no failing rsyncs noted). Any suggesting about how to get more information about what went wrong e.g: "10.11.18.67": {"obj08": 2348}, how to find what those 2348 errors were? regards Mark P.s: basic sanity checking is ok - uploaded objects go where they should and can be retrieved for 2.7 or 2.23 servers ok (the old and new version servers agree about object placement)

2 4