September 2020 - openstack-discuss

[Kolla Ansible] Error in "heat" stage during deployment - Followed Quick Start Guide (ubuntu 18.04.5, virtualbox)
by Lukasluedke＠web.de 06 Sep '20

06 Sep '20

Hi everyone, I am new to openstack and just followed the "Quick Start" guide for kolla-ansible [https://docs.openstack.org/kolla-ansible/ussuri/user/quickstart.html] on ussuri release, but now having some problems during deployment. I used virtualbox to create a new vm with ubuntu 18.04.5 [https://releases.ubuntu.com/18.04/ubuntu-18.04.5-live-server-amd64.iso.torr…] and attached one interface (enp0s3, 10.0.2.15) with Nat-Network and the second interface as for the neutron interface (enp0s8, no ip) an internal network and named it "neutron". Then I followed the guide to create a new virtual python environment (logged in over ssh using "user" user). -- sudo apt-get update sudo apt-get install python3-dev libffi-dev gcc libssl-dev sudo apt-get install python3-venv python3 -m venv $(pwd) source $(pwd)/bin/activate pip install -U pip pip install 'ansible<2.10' -- Everything seems to work as mentioned with no problems so far. (Beside the missing "python-venv" package in the docs) -- pip install kolla-ansible sudo mkdir -p /etc/kolla sudo chown $USER:$USER /etc/kolla cp -r $(pwd)/share/kolla-ansible/etc_examples/kolla/* /etc/kolla cp $(pwd)/share/kolla-ansible/ansible/inventory/* . mkdir /etc/ansible sudo nano /etc/ansible/ansible.cfg ### content from /etc/ansible/ansible.cfg [defaults] host_key_checking=False pipelining=True forks=100 ### -- I then modified the "multinode" file to only deploy on localhost (replaced all control01, etc. with localhost) as I first wanted to try openstack on one node. The ping worked, changed the globals.yml file according to the documentation, then bootstrap and the prechecks worked so far with no issue. -- ansible -i multinode all -m ping kolla-genpwd sudo nano /etc/kolla/globals.yml ### additional content of /etc/kolla/globals.yml kolla_base_distro: "ubuntu" kolla_install_type: "source" network_interface: "enp0s3" neutron_external_interface: "enp0s8" kolla_internal_vip_address: "10.0.2.10" ### kolla-ansible -i ./multinode bootstrap-servers kolla-ansible -i ./multinode prechecks -- But on the deploy stage I am stuck right now: -- kolla-ansible -i ./multinode deploy -- The mentioned error is telling me, that the keystone service is not available. I attached the error log file, because the error is longer. As a side node, now when I try to login using tty, the vm is nearly frozen and only prompts very, very slowly (about 10 minutes for prompt). ssh in putty is also frozen and the connection broke after about 30 minutes. Does anyone know what is wrong or has some hints/insight on how to correctly deploy openstack using kolla-ansible? Best Regards, Lukas Lüdke

2 2

how-to enable s3api
by admin 05 Sep '20

05 Sep '20

hi where is there a normal manual on how to enable s3api support in swift? trying to go through https://docs.openstack.org/swift/latest/middleware.html and as a result keystone gives an error in authorization (trying to connect via (s3curl) Authorization failed. The request you have made requires authentication. from x.x.x.x Unauthorized: The request you have made requires authentication.

2 1

[mq][mariadb] what kind of reasons are considered when choosing the mq/mariadb version
by Sheldon Hu(胡玉鹏) 05 Sep '20

05 Sep '20

Hi all I use kolla to build the openstack version, use kolla-ansible to deploy rabbitmq is v3.7.10, mariadb is v10.1.x when building rocky version rabbitmq is v3.8.5, mariadb is v10.3.x when building ussuri version what kind of reasons are considered as rabbitmq version is changed from v3.7.10 to v3.8.5 ? what kind of reasons are considered as mariadb version is changed from v10.1.x to v10.3.x ? many thanks. Sheldon Hu | 胡玉鹏 CBRD | 云计算与大数据研发部 T: 18663761785 E: <mailto:huyp@inspur.com> huyp(a)inspur.com 浪潮电子信息产业股份有限公司 Inspur Electronic Information Industry Co.,Ltd. 山东省济南市历城区东八区企业公馆A7-1 Building A7-1, Dongbaqu Office Block, Licheng District, Jinan,Shandong Province, PRC 浪潮云海

2 1

[scientific]
by Foundjem Armstrong 05 Sep '20

05 Sep '20

Hello Blair, I am wondering of this email is still functional. Regards, Armstrong

2 1

[TC] Looking for a volunteer to help with community goal selection
by Nate Johnston 05 Sep '20

05 Sep '20

TC members, Since I have not been able to get in contact with Graham, I would like to see if there is anyone who would like to work with me for the selection of community goals for the W series. The commitment would be to identify a few good candidates that have well formulated goal proposals and champions, which usually involves a meeting followed by some reaching out and logistical work. I am hoping to increase the pool of those familiar with the goal selection process, so this is a great place to jump in if you have not done this before. Also, if Graham does have time then we can make sure to include him as well. Two is good but three is better! Please let me know if you are interested. Thanks! Nate

1 0

Mentoring Boston University Students
by Kendall Nelson 05 Sep '20

05 Sep '20

Hello! As you may or may not know, the last two years various community members have mentored students from North Dakota State University for a semester to work on projects in OpenStack. Recently, I learned of a similar program at Boston University and they are still looking for mentors interested for the upcoming semester. Essentially you would have 5 to 7 students for 13 weeks to mentor and work on some feature or effort in your project. The time to apply is running out however as the deadline is Sept 3rd. If you are interested, please let me know ASAP! I am happy to help get the students up to speed with the community and getting their workspaces set up, but the actual work they would do is more up to you :) -Kendall (diablo_rojo)

2 3

[tripleo] centos-binary -> openstack-
by Wesley Hayutin 05 Sep '20

05 Sep '20

Greetings, The container names in master have changed from centos-binary* to openstack*. https://opendev.org/openstack/tripleo-common/src/branch/master/container-im… https://opendev.org/openstack/tripleo-common/commit/90f6de7a7fab15e9161c1f0… If your patches are failing to pull https://registry-1.docker.io/v2/tripleomaster/centos-binary* it's not going to be fixed in a recheck. Check that your patches are rebased and your dependent patches are rebased. Thanks 0/

4 3

[oslo][ffe] request for oslo
by Herve Beraud 05 Sep '20

05 Sep '20

Hey Oslofolk, I request an FFE for these oslo.messaging changes [1]. The goal of these changes is to run rabbitmq heartbeat in python thread by default. Also these changes deprecating this option to prepare future removal and force to always run heartbeat in a python thread whatever the context. Land these changes during the victoria cycle can help us to prime the option removal during the next cycle. Thanks for your time, [1] https://review.opendev.org/#/c/747395/ -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

3 3

[all][TC] OpenStack Client (OSC) vs python-*clients
by Belmiro Moreira 04 Sep '20

04 Sep '20

Hi, during the last PTG the TC discussed the problem of supporting different clients (OpenStack Client - OSC vs python-*clients) [1]. Currently, we don't have feature parity between the OSC and the python-*clients. Different OpenStack projects invest in different clients. This can be a huge problem for users/ops. Depending on the projects deployed in their infrastructures, they need to use different clients for different tasks. It's confusing because of the partial implementation in the OSC. There was also the proposal to enforce new functionality only in the SDK (and optionally the OSC) and not the project’s specific clients to stop increasing the disparity between the two. We would like to understand first the problems and missing pieces that projects are facing to move into OSC and help to overcome them. Let us know. Belmiro, on behalf of the TC [1] http://lists.openstack.org/pipermail/openstack-discuss/2020-June/015418.html

13 20

[nova] Session Recording for Feedback
by Apsey, Christopher 04 Sep '20

04 Sep '20

Nova supports using the websockify -record functionality to capture frames sent over arbitrary console sessions. Currently, the record = path/to/file option in nova.conf saves these sessions on nova-(spice|vnc)proxy endpoints at the specified path in the format of /path/to/file.(session number). These session numbers appear to be incrementally created starting from 1 and don't really have an association with their respective instance from what I can tell. It would be useful to have these files saved using the instance UUID instead. Ignoring that minor problem, playing back these files is a challenge. I've tried using noVNCs vnc_playback.html function from both the master branch and stable/v0.6 with no luck - it looks like the libraries and the utilities haven't been maintained at the same cadence and there are missing functions that the player is expecting. My goal here is to be able to capture the activities of students as they progress through various exercises and then have instructors be able to go over their work with them after the fact if they have questions. Has anyone been able to do this (or something like this) successfully, or are we just better off trying to do this in-band per instance? Chris Apsey GEORGIA CYBER CENTER

2 1

Re: Kuryr openstack
by mahdi abbasi 04 Sep '20

04 Sep '20

Thanks a lot This issue has been resolved On Thu, 3 Sep 2020, 23:28 Radosław Piliszek, <radoslaw.piliszek(a)gmail.com> wrote: > Hi Mahdi, > > have you created any networks (as in Neutron resource) in that project? > You need to have at least one network for Zun to use. > > -yoctozepto > > On Thu, Sep 3, 2020 at 6:50 PM mahdi abbasi <mahdi.abbasi.2013(a)gmail.com> > wrote: > > > > Hi development team, > > > > Do i need a special configuration to use kuryr when using openvswitch? > > I get the following error when starting Docker container in zun: > > > > Unable to create the network.No tenant network is availble for > allocation. > > > > Please help me. > > > > Best regards > > Mahdi >

1 0

Kuryr openstack
by mahdi abbasi 04 Sep '20

04 Sep '20

Hi development team, Do i need a special configuration to use kuryr when using openvswitch? I get the following error when starting Docker container in zun: Unable to create the network.No tenant network is availble for allocation. Please help me. Best regards Mahdi

2 2

[InteropWG] Brainstorming and looking for Ideas from cross project teams
by prakash RAMCHANDRAN 04 Sep '20

04 Sep '20

Hi all, Interop WG i seeking cross platform participation to enable Baremetal and Kubernetes ready Modules in OpenStack & Open Infrastructure. Please add what you would like to get help from Interop Group and how in turn you can help Interop WG goals of enabling Multi-Cloud & Hybrid Cloud Logo Programs to be initiated in CY Q4 2019. We like to discuss and get any new ideas whetted and supported by community. Please add and join us in Forum, PTG and Summit. https://etherpad.opendev.org/p/2020-Wallaby-interop-brainstorming Look forward to your Projects valued participation in Interop Branding efforts.Our next Interop Call is shceduled for Sept 11th 10 AM PDT, please add before that all you can bring to table. For Interop WG Prakash RamchandranIndi Member BoD

1 0

[oslo] Feature Freeze
by Ben Nemec 04 Sep '20

04 Sep '20

This is just a heads up that Oslo is now in feature freeze. Any changes that would require a feature release of an Oslo library will need to request an FFE if they want to make the Victoria release. Please send an email tagged "[oslo][ffe]" to the list to make such a request. I expect we'll branch the Oslo repos in the near future, so some feature development can continue then. However, we generally prefer not to merge invasive changes between feature freeze and release just in case a last-minute bugfix backport is needed. Disentangling a bug fix from a major feature could be a chore. Thanks. -Ben

1 0

[neutron] Drivers meeting - 04.09.2020
by Slawek Kaplonski 04 Sep '20

04 Sep '20

Hi, As there is no agenda for tomorrow's meeting, I will cancel it. Lets focus on reviewing patches related to the RFE scheduled for Victoria-3: https://launchpad.net/neutron/+milestone/victoria-3 See You all online and have a great weekend o/ -- Slawek Kaplonski Principal software engineer Red Hat

1 0

Issue with libvirt unable to kill processes
by Grant Morley 04 Sep '20

04 Sep '20

Hi All, I was wondering if anyone has come across an issue with libvirt seemingly having an issue with instances all of a sudden "locking up" with the following error: Failed to terminate process 2263874 with SIGKILL: Device or resource busy In the nova logs I am seeing: 2020-09-03 09:13:43.208 2659995 INFO nova.compute.manager [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Rebooting instance 2020-09-03 09:15:52.429 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Failed to soft reboot instance. Trying hard reboot. 2020-09-03 09:16:32.450 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 2263874 with SIGKILL: Device or resource busy; attempt 1 of 3: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:12.484 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 2263874 with SIGKILL: Device or resource busy; attempt 2 of 3: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:52.516 2659995 WARNING nova.virt.libvirt.driver [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Error from libvirt during destroy. Code=38 Error=Failed to terminate process 2263874 with SIGKILL: Device or resource busy; attempt 3 of 3: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:52.526 2659995 ERROR nova.compute.manager [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Cannot reboot instance: Failed to terminate process 2263874 with SIGKILL: Device or resource busy: libvirtError: Failed to terminate process 2263874 with SIGKILL: Device or resource busy 2020-09-03 09:17:53.026 2659995 INFO nova.compute.manager [req-7ffb9b7c-799b-40dc-be04-598bfda2e2fc 6ddb647baf9343b09d7f8f7a32b0b43d 8cce5e1532e6435a90b168077664bbdf - default default] [instance: f3a8c916-28f5-432d-9b8e-c3056d2dee5a] Successfully reverted task state from reboot_started on failure for instance. It seems to be caused when a reboot happens to an instance. If you reset the state and try again, the same error occurs. You also seemingly cannot kill off any libvirt process that is attached to that instance. To me it looks like it could be a kernel issue with libvirt but I could be wrong? Does anyone know of a workaround for this other than maybe restarting a compute host? Many thanks,

2 1

Kuryr openstack
by mahdi abbasi 04 Sep '20

04 Sep '20

Hi development team, Do i need a special configuration to use kuryr when using openvswitch? I get the following error when starting Docker container in zun. Please help me. Best regards Mahdi

1 0

[openstack-ansible] OpenStack Ansible deployment fails due to lxc containers not having network connection
by Oliver Wenz 03 Sep '20

03 Sep '20

I'm trying to deploy OpenStack Ansible. When running the first playbook ```openstack-ansible setup-hosts.yml```, there are errors for all containers during the task ```[openstack_hosts : Remove the blacklisted packages]``` (see below) and the playbook fails. ``` fatal: [infra1_repo_container-1f1565cd]: FAILED! => {"changed": false, "cmd": "apt-get update", "msg": "E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic Release' no longer has a Release file. E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-updates Release' no longer has a Release file. E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-backports Release' no longer has a Release file. E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-security Release' no longer has a Release file.", "rc": 100, "stderr": "E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic Release' no longer has a Release file. E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-updates Release' no longer has a Release file. E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-backports Release' no longer has a Release file. E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-security Release' no longer has a Release file. ", "stderr_lines": ["E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic Release' no longer has a Release file.", "E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-updates Release' no longer has a Release file.", "E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-backports Release' no longer has a Release file.", "E: The repository 'http://ubuntu.mirror.lrz.de/ubuntu bionic-security Release' no longer has a Release file."], "stdout": "Ign:1 http://ubuntu.mirror.lrz.de/ubuntu bionic InRelease Ign:2 http://ubuntu.mirror.lrz.de/ubuntu bionic-updates InRelease Ign:3 http://ubuntu.mirror.lrz.de/ubuntu bionic-backports InRelease Ign:4 http://ubuntu.mirror.lrz.de/ubuntu bionic-security InRelease Err:5 http://ubuntu.mirror.lrz.de/ubuntu bionic Release Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable) Err:6 http://ubuntu.mirror.lrz.de/ubuntu bionic-updates Release Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable) Err:7 http://ubuntu.mirror.lrz.de/ubuntu bionic-backports Release Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable) Err:8 http://ubuntu.mirror.lrz.de/ubuntu bionic-security Release Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable) Reading package lists... ", "stdout_lines": ["Ign:1 http://ubuntu.mirror.lrz.de/ubuntu bionic InRelease", "Ign:2 http://ubuntu.mirror.lrz.de/ubuntu bionic-updates InRelease", "Ign:3 http://ubuntu.mirror.lrz.de/ubuntu bionic-backports InRelease", "Ign:4 http://ubuntu.mirror.lrz.de/ubuntu bionic-security InRelease", "Err:5 http://ubuntu.mirror.lrz.de/ubuntu bionic Release", " Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable)", "Err:6 http://ubuntu.mirror.lrz.de/ubuntu bionic-updates Release", " Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable)", "Err:7 http://ubuntu.mirror.lrz.de/ubuntu bionic-backports Release", " Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable)", "Err:8 http://ubuntu.mirror.lrz.de/ubuntu bionic-security Release", " Cannot initiate the connection to 192.168.100.6:8000 (192.168.100.6). - connect (101: Network is unreachable)", "Reading package lists..."]} ``` When I attach to any container and run ```ping 192.168.100.6``` (local DNS), I get the same error (```connect: Network is unreachable```). However, when I specify an interface by running ```ping -I eth1 192.168.100.6``` there is a successful connection. Running ```ip r``` on the infra_cinder container yields: ``` 10.0.3.0/24 dev eth2 proto kernel scope link src 10.0.3.5 192.168.110.0/24 dev eth1 proto kernel scope link src 192.168.110.232 ``` so there seems to be no default route which is why the connection fails (similar for the other infra containers). Shouldn't OSA automatically configure this? I didn't find anything regarding a default route on containers in the Docs. Here's my openstack_user_config.yml: ``` cidr_networks: container: 192.168.110.0/24 tunnel: 192.168.32.0/24 storage: 10.0.3.0/24 used_ips: - "192.168.110.1,192.168.110.2" - "192.168.110.111" - "192.168.110.115" - "192.168.110.117,192.168.110.118" - "192.168.110.131,192.168.110.140" - "192.168.110.201,192.168.110.207" - "192.168.32.1,192.168.32.2" - "192.168.32.201,192.168.32.207" - "10.0.3.1" - "10.0.3.11,10.0.3.14" - "10.0.3.21,10.0.3.24" - "10.0.3.31,10.0.3.42" - "10.0.3.201,10.0.3.207" global_overrides: # The internal and external VIP should be different IPs, however they # do not need to be on separate networks. external_lb_vip_address: 192.168.100.168 internal_lb_vip_address: 192.168.110.201 management_bridge: "br-mgmt" provider_networks: - network: container_bridge: "br-mgmt" container_type: "veth" container_interface: "eth1" ip_from_q: "container" type: "raw" group_binds: - all_containers - hosts is_container_address: true - network: container_bridge: "br-vxlan" container_type: "veth" container_interface: "eth10" ip_from_q: "tunnel" type: "vxlan" range: "1:1000" net_name: "vxlan" group_binds: - neutron_linuxbridge_agent - network: container_bridge: "br-ext1" container_type: "veth" container_interface: "eth12" host_bind_override: "eth12" type: "flat" net_name: "ext_net" group_binds: - neutron_linuxbridge_agent - network: container_bridge: "br-storage" container_type: "veth" container_interface: "eth2" ip_from_q: "storage" type: "raw" group_binds: - glance_api - cinder_api - cinder_volume - nova_compute - swift-proxy ### ### Infrastructure ### # galera, memcache, rabbitmq, utility shared-infra_hosts: infra1: ip: 192.168.110.201 # repository (apt cache, python packages, etc) repo-infra_hosts: infra1: ip: 192.168.110.201 # load balancer haproxy_hosts: infra1: ip: 192.168.110.201 ### ### OpenStack ### os-infra_hosts: infra1: ip: 192.168.110.201 identity_hosts: infra1: ip: 192.168.110.201 network_hosts: infra1: ip: 192.168.110.201 compute_hosts: compute1: ip: 192.168.110.204 compute2: ip: 192.168.110.205 compute3: ip: 192.168.110.206 compute4: ip: 192.168.110.207 storage-infra_hosts: infra1: ip: 192.168.110.201 storage_hosts: lvm-storage1: ip: 192.168.110.202 container_vars: cinder_backends: lvm: volume_backend_name: LVM_iSCSI volume_driver: cinder.volume.drivers.lvm.LVMVolumeDriver volume_group: cinder_volumes iscsi_ip_address: "{{ cinder_storage_address }}" limit_container_types: cinder_volume ``` I also asked this question on the server fault stackexchange: https://serverfault.com/questions/1032573/openstack-ansible-deployment-fail… Kind regards, Oliver

2 1

[API SIG] Ending office hours for the API-SIG
by Michael McCune 03 Sep '20

03 Sep '20

hello all, the API-SIG has held meetings and office hours for several years now. since our migration from a weekly meeting to an office hour we have monitored a continual decrease in attendance for these hours, aside from the core membership. after discussion over the course of the last month or two, we have agreed to end the API-SIG office hours. we will hold office hours this week as our final meeting. if you need to contact the SIG in the future, we are still available in the #openstack-sdks channel on freenode as well as here on the mailing list. in the future we would like to see continued migration of the API-SIG closer to the SDKs group. these two groups will essentially become a singular place to discuss all things related to SDKs and APIs in OpenStack. thank you all for the years of participation and discussion that helped us as a community to define a set of best practices and guidelines for API work in OpenStack. peace o/

1 0

senlin auto scaling question
by Satish Patel 03 Sep '20

03 Sep '20

Folks, I have created very simple cluster using following command openstack cluster create --profile myserver --desired-capacity 2 --min-size 2 --max-size 3 --strict my-asg It spun up 2 vm immediately now because the desired capacity is 2 so I am assuming if any node dies in the cluster it should spin up node to make count 2 right? so i killed one of node with "nove delete <instance-foo-1>" but senlin didn't create node automatically to make desired capacity 2 (In AWS when you kill node in ASG it will create new node so is this senlin different then AWS?)

3 5

[kolla] questions when using external mysql
by sosogh 03 Sep '20

03 Sep '20

Hi list: I want to use kolla-ansible to deploy openstack , but using external mysql. I am following these docs: https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html https://docs.openstack.org/kolla-ansible/latest/reference/databases/externa…. I have some questions: ################ ## Question 1 ## ################ According to the offical doc , if setting it in inventory file(multinode), kolla-ansible -i ./multinode deploy will throw out error: I guest when kolla-ansible running the playbook against myexternalmariadbloadbalancer.com , the """register: find_custom_fluentd_inputs""" in """TASK [common : Find custom fluentd input config files]""" maybe null . ################ ## Question 2 ## ################ According to the offical doc , If the MariaDB username is not root, set database_username in /etc/kolla/globals.yml file: But in kolla-ansible/ansible/roles/xxxxxx/tasks/bootstrap.yml , they use ''' login_user: "{{ database_user }}" ''' ， for example : So at last , I took the following steps: 1. """not""" setting [mariadb] in inventory file(multinode) 2. set "database_user: openstack" for "privillegeduser" PS: My idea is that if using an external ready-to-use mysql (cluster), it is enough to tell kolla-ansible only the address/user/password of the external DB. i.e. setting them in the file /etc/kolla/globals.yml and passwords.yml , no need to add it into inventory file(multinode) Finally , it is successful to deploy openstack via kolla-ansible . So far I have not found any problems. Are the steps what I took good ( enough ) ? Thank you !

2 4

[tacker] Wallaby vPTG
by Yasufumi Ogawa 03 Sep '20

03 Sep '20

Hi team, I booked our slots, 27-29 Oct 6am-8am UTC, for the next vPTG[1] as we agreed in previous irc meeting. I also prepared an etherpad [2], so please add your name and suggestions. [1] https://ethercalc.openstack.org/7xp2pcbh1ncb [2] https://etherpad.opendev.org/p/Tacker-PTG-Wallaby Thanks, Yasufumi

2 2

[ovn][neutron][ussuri] How to configure for overlay in DPDK with OVN ?
by 한승진 03 Sep '20

03 Sep '20

Hi, I’m trying to implement dpdk-ovs with ovn for our openstack environment. I create a bond interface(tunbound) with ovs and tun0 and tun1 are slave interfaces of the bond interface named tunbond. Bridge br-int fail_mode: secure datapath_type: netdev Port tunbond Interface tun0 type: dpdk options: {dpdk-devargs="0000:1a:00.1", n_rxq="2"} Interface tun1 type: dpdk options: {dpdk-devargs="0000:d8:00.1", n_rxq="2"} Port br-int Interface br-int type: internal Now, I need to configure remote-ovn options to the ovs. *I am confused how can I define the IP address of the bonded interface with ovs?* Theses are commands for connect remote ovn services. ovs-vsctl set open . external-ids:ovn-remote=tcp:{controller ip}:6642 ovs-vsctl set open . external-ids:ovn-encap-type=geneve *1* This below command make it me confused. ovs-vsctl set open . external-ids:ovn-encap-ip={local ip} How should I resolve this issue? Regards, John Haan

2 2

[tripleo, ironic] Error: Could not retrieve ... pxelinux.0
by Pete Zaitcev 03 Sep '20

03 Sep '20

Hello: I wanted to give the TripleO a try, so started follow our installation guide for Ussuri, and eventually made it to "openstack undercloud install". It fails with something like this: Aug 28 10:10:53 undercloud puppet-user[48657]: Error: /Stage[main]/Ironic::Pxe/File[/var/lib/ironic/tftpboot/ipxe.efi]: Could not evaluate: Could not retrieve information from environment production source(s) file:/usr/share/ipxe/ipxe-x86_64.efi Aug 27 20:05:42 undercloud puppet-user[37048]: Error: /Stage[main]/Ironic::Pxe/Ironic::Pxe::Tftpboot_file[pxelinux.0]/File[/var/lib/ironic/tftpboot/pxelinux.0]: Could not evaluate: Could not retrieve information from environment production source(s) file:/tftpboot/pxelinux.0 Does anyone have an idea what it wants? I added a couple of packages on the host system that provided the files mentioned in the message, but it made no difference. Ussuri is conteinerized anyway. Since I'm very new to this, I have no clue where to look at all. The nearest task is a wrapper of some kind, so the install-undercloud.log looks like this: 2020-08-28 14:11:31.397 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] TASK [Run container-puppet tasks (generate config) during step 1 with paunch] *** 2020-08-28 14:11:31.397 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Friday 28 August 2020 14:11:31 -0400 (0:00:00.302) 0:06:28.734 ********* 2020-08-28 14:11:32.223 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] changed: [undercloud] 2020-08-28 14:11:32.325 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] 2020-08-28 14:11:32.326 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] TASK [Wait for container-puppet tasks (generate config) to finish] ************* 2020-08-28 14:11:32.326 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] Friday 28 August 2020 14:11:32 -0400 (0:00:00.928) 0:06:29.663 ********* 2020-08-28 14:11:32.948 60599 WARNING tripleoclient.v1.tripleo_deploy.Deploy [ ] WAITING FOR COMPLETION: Wait for container-puppet tasks (generate config) to finish (1200 retries left). . . . If anyone could tell roughly what is supposed to be going on here, it would be great. I may be able figure out the rest. Greetings, -- Pete

2 3

Re: [neutron][ovn] OVN Performance
by Apsey, Christopher 02 Sep '20

02 Sep '20

All, Just wanted to loop back here and give an update. For reference, [1] (blue means successful action, red means failed action) is the result we got when booting 5000 instances in rally [2] before the Red Hat OVN devs poked around inside our environment, and [3] is the result after. The differences are obviously pretty significant. I think the biggest change was setting metadata_workers = 2 in neutron_ovn_metadata_agent.ini on the compute nodes per https://bugs.launchpad.net/neutron/+bug/1893656. We have 64C/128T on all compute nodes, so the default neutron calculation of scaling metadata workers based on available cores created 900+ connections to the southbound db at idle; after the control plane got loaded up it just quit around 2500 instances (my guess is it hit the open file limit, although I don’t think increasing it would have made it better for much longer since the number of connections were increasing exponentially). Capping the number of metadata workers decreased open southbound connections by 90%. Even more telling was that rally was able to successfully clean up after itself after we made that change, whereas previously it wasn’t even able to successfully tear down any of the instances that were made, indicating that the control plane was completely toast. Note that the choppiness towards the end of [3] had nothing to do with OVN – our compute nodes had a loadavg approaching 1000 at that point, so they were just starved for cpu cycles. This would have scaled even better with additional compute nodes. The other piece was RAFT. Currently, RDO is shipping with ovs 2.12, but 2.13 has a bunch of RAFT fixes in it that improve stability and knock out some bugs. We were having issues with chassis registration on 2.12, but after using the 2.13 package from cbs, all those issues went away. Big thanks to the great people at Red Hat on the cc line for volunteering their valuable time to take a look. I’m now significantly more comfortable with defaulting to OVN as the backend of choice as the performance delta is now gone. That said, should the community consider dropping linuxbridge as the backend in the official upstream docs and jump straight to OVN rather than ml2/OVS? I think that would increase the test base and help shine light on other issues as time goes on. My org can devote some time to doing this work if the community agrees that it’s the right action to take. Hope that’s helpful! [1] https://ibb.co/GTjZP2y [2] https://pastebin.com/5pEDZ7dY [3] https://ibb.co/pfB9KTV Chris Apsey GEORGIA CYBER CENTER From: Apsey, Christopher Sent: Thursday, August 27, 2020 11:33 AM To: Assaf Muller <amuller(a)redhat.com> Cc: openstack-discuss(a)lists.openstack.org; Lucas Alvares Gomes Martins <lmartins(a)redhat.com>; Jakub Libosvar <jlibosva(a)redhat.com>; Daniel Alvarez Sanchez <dalvarez(a)redhat.com> Subject: RE: [EXTERNAL] Re: [neutron][ovn] OVN Performance Assaf, We can absolutely support engineering poking around in our environment (and possibly an even larger one at my previous employer that was experiencing similar issues during testing). We can take this offline so we don’t spam the mailing list. Just let me know how to proceed, Thanks! Chris Apsey GEORGIA CYBER CENTER From: Assaf Muller <amuller(a)redhat.com<mailto:amuller@redhat.com>> Sent: Thursday, August 27, 2020 11:18 AM To: Apsey, Christopher <CAPSEY(a)augusta.edu<mailto:CAPSEY@augusta.edu>> Cc: openstack-discuss(a)lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>; Lucas Alvares Gomes Martins <lmartins(a)redhat.com<mailto:lmartins@redhat.com>>; Jakub Libosvar <jlibosva(a)redhat.com<mailto:jlibosva@redhat.com>>; Daniel Alvarez Sanchez <dalvarez(a)redhat.com<mailto:dalvarez@redhat.com>> Subject: [EXTERNAL] Re: [neutron][ovn] OVN Performance CAUTION: EXTERNAL SENDER This email originated from an external source. Please exercise caution before opening attachments, clicking links, replying, or providing information to the sender. If you believe it to be fraudulent, contact the AU Cybersecurity Hotline at 72-CYBER (2-9237 / 706-722-9237) or 72CYBER(a)augusta.edu<mailto:72CYBER@augusta.edu> The most efficient way about this is to give one or more of the Engineers working on OpenStack OVN upstream (I've added a few to this thread) temporary access to an environment that can reproduce issues you're seeing, we could then document the issues and work towards solutions. If that's not possible, if you could provide reproducer scripts, or alternatively sharpen the reproduction method, we'll take a look. What you've described is not something that's 'acceptable', OVN should definitely not scale worse than Neutron with the Linux Bridge agent. It's possible that the particular issues you ran in to is something that we've already seen internally at Red Hat, or with our customers, and we're already working on fixes in future versions of OVN - I can't tell you until you elaborate on the details of the issues you're seeing. In any case, the upstream community is committed to improving OVN scale and fixing scale issues as they pop up. Coincidentally, Red Hat scale engineers just published an article [1] about work they've done to scale RH-OSP 16.1 (== OpenStack Train on CentOS 8, with OVN 2.13 and TripleO) to 700 compute nodes. [1] https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-…<https://www.redhat.com/en/blog/scaling-red-hat-openstack-platform-161-more-…> On Thu, Aug 27, 2020 at 10:44 AM Apsey, Christopher <CAPSEY(a)augusta.edu<mailto:CAPSEY@augusta.edu>> wrote: > > All, > > > > I know that OVN is going to become the default neutron backend at some point and displace linuxbridge as the default configuration option in the docs, but we have noticed a pretty significant performance disparity between OVN and linuxbridge on identical hardware over the past year or so in a few different environments[1]. I know that example is unscientific, but similar results have been borne out in many different scenarios from what we have observed. There are three main problems from what we see: > > > > 1. OVN does not handle large concurrent requests as well as linuxbridge. Additionally, linuxbridge concurrent capacity grows (not linearly, but grows nonetheless) by adding additional neutron API endpoints and RPC agents. OVN does not really horizontally scale by adding additional API endpoints, from what we have observed. > > 2. OVN gets significantly slower as load on the system grows. We have observed a soft cap of about 2000-2500 instances in a given deployment before ovn-backed neutron stops responding altogether to nova requests (even for booting a single instance). We have observed linuxbridge get to 5000+ instances before it starts to struggle on the same hardware (and we think that linuxbridge can go further with improved provider network design in that particular case). > > 3. Once the southbound database process hits 100% CPU usage on the leader in the ovn cluster, it’s game over (probably causes 1+2) > > > > It's entirely possible that we just don’t understand OVN well enough to tune it [2][3][4], but then the question becomes how do we get that tuning knowledge into the docs so people don’t scratch their heads when their cool new OVN deployment scales 40% as well as their ancient linuxbridge-based one? > > > > If it is ‘known’ that OVN has some scaling challenges, is there a plan to fix it, and what is the best way to contribute to doing so? > > > > We have observed similar results on Ubuntu 18.04/20.04 and CentOS 7/8 on Stein, Train, and Ussuri. > > > > [1] https://pastebin.com/kyyURTJm<https://pastebin.com/kyyURTJm> > > [2] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb<https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/ovsdb> > > [3] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron<https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/neutron> > > [4] https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute<https://github.com/GeorgiaCyber/kinetic/tree/master/formulas/compute> > > > > Chris Apsey > > GEORGIA CYBER CENTER > >

2 1

[manila] Victoria Collab Review next Tuesday (Sep 1st)
by Douglas 02 Sep '20

02 Sep '20

Hi everybody We will have a new edition of our collaborative review next Tuesday, September 1st, where we'll go through the code and review the proposed feature Share Server Migration[1][2]. This meeting is scheduled for two hours, starting at 5:00PM UTC. Meeting notes and videoconference links will be available here[3]. Feel free to attend if you are interested and available. Hoping to see you there, - dviroel [1] https://opendev.org/openstack/manila-specs/src/branch/master/specs/victoria… [2] https://review.opendev.org/#/q/topic:bp/share-server-migration+(status:open) [3] https://etherpad.opendev.org/p/manila-victoria-collab-review

1 1

Graceful stopping of RabbitMQ AMQP notification listener
by Anil Jangam 02 Sep '20

02 Sep '20

Hi, I have coded OpenStack AMQP listener following the example and it is working fine. https://github.com/gibizer/nova-notification-demo/blob/master/ws_forwarder.… The related code snippets of the NotificationHandler class are shown as follows. # Initialize the AMQP listener def init(self, cluster_ip, user, password, port): cfg.CONF() cluster_url = "rabbit://" + user + ":" + password + "@" + cluster_ip + ":" + port + "/" transport = oslo_messaging.get_notification_transport(cfg.CONF, url=cluster_url) targets = [ oslo_messaging.Target(topic='versioned_notifications'), ] endpoints = [self.__endpoint] # Initialize the notification listener try: self.__amqp_listener = oslo_messaging.get_notification_listener(transport, targets, endpoints, executor='threading') except NotImplementedError as err: LOGGER.error("Failed to initialize the notification listener {}".format(err)) return False LOGGER.debug("Initialized the notification listener {}".format(cluster_url)) return True # Arm the compute event listeners def start_amqp_event_listener(self): # Start the notification handler LOGGER.debug("Started the OpenStack notification handler") self.__amqp_listener.start() # Disarm the compute event listeners def stop_amqp_event_listener(self): LOGGER.debug("Stopping the OpenStack notification handler") if self.__amqp_listener is not None: self.__amqp_listener.stop() I am using this interface from a new process handler function, however, when I invoke the stop_amqp_eent_listener() method, my process hangs. It does not terminate naturally. I verified that the self.__amqp_listener.stop() function is not returning. Is there anything missing in this code? Is there any specific consideration when calling the listener from a new process? Can someone provide a clue? # Stop the worker def stop(self): # Stop the AMQP notification handler self.__amqp_handler.stop_amqp_event_listener() LOGGER.debug("Stopped the worker for {}".format(self.__ops_conn_info.cluster_ip)) /anil.

1 1

[Magnum][Kayobe] Magnum Kubernetes clusters failing to be created (bugs?)
by Tony Pearce 02 Sep '20

02 Sep '20

Hi guys, I hope you are all keeping safe and well at the moment. I am trying to launch Kubernetes clusters into Openstack Train which has been deployed via Kayobe (Kayobe as I understand is a wrapper for kolla-ansible). There have been a few strange issues here and I've struggled to isolate them. These issues started recently after a fresh Openstack deployment some months ago (around February 2020) to give some context. This Openstack is not "live" as I've been trying to get to the bottom of the issues: Issue 1. When trying to launch a cluster we get error "Resource Create Failed: Forbidden: Resources.Kube Masters.Resources[0].Resources.Kube-Master: Only Volume-Backed Servers Are Allowed For Flavors With Zero Disk. " Issue 2. After successfully creating a cluster of a smaller node size, the "resize cluster" is failing (however update the cluster is working). Some background on this specific environment: Deployed via Kayobe, with these components: Cinder, Designate, iscsid, Magnum, Multipathd, neutron provider networks The Cinder component integrates with iSCSI SAN storage using the Nimble driver. This is the only storage. In order to prevent Openstack from allocating Compute node local HDD as instance storage, I have all flavours configured with root disk / ephemeral disk / swap disk = "0MB". This then results in all instance data being stored on the backend Cinder storage appliance. I was able to get a cluster deployed by first creating the template as needed, then when launching the cluster Horizon prompts you for items already there in the template such as number of nodes, node flavour and labels etc. I re-supplied all of the info (as to duplicate it) and then tried creating the cluster. After many many times trying over the course of a few weeks to a few months it was successful. I was then able to work around the issue #2 above to get it increased in size. When looking at the logs for issue #2, it looks like some content is missing in the API but I am not certain. I will include a link to the pastebin below [1]. When trying to resize the cluster, Horizon gives error: "Error: Unable to resize given cluster id: 99693dbf-160a-40e0-9ed4-93f3370367ee". I then searched the controller node /var/log directory for this ID and found "horizon.log [:error] [pid 25] Not Found: /api/container_infra/clusters/99693dbf-160a-40e0-9ed4-93f3370367ee/resize". Going to the Horizon menu "update cluster" allows you to increase the number of nodes and then save/apply the config which does indeed resize the cluster. Regarding issue #1, we've been unable to deploy a cluster in a new project and the error is hinting it relates to the flavours having 0MB disk specified, though this error is new and we've been successful previously with deploying clusters (albeit with the hit-and-miss experiences) using the flavour with 0MB disk as described above. Again I searched for the (stack) ID after the failure, in the logs on the controller and I obtained not much more than the error already seen with Horizon [2]. I was able to create new flavours with root disk = 15GB and then successfully deploy a cluster on the next immediate try. Update cluster from 3 nodes to 6 nodes was also immediately successful. However I see the compute nodes "used" disk space increasing after increasing the cluster size which is an issue as the compute node has very limited HDD capacity (32GB SD card). At this point I also checked 1) previously installed cluster using the 0MB disk flavour and 2) new instances using the 0MB disk flavour. I notice that the previous cluster is having host storage allocated but while the new instance is not having host storage allocated. So the cluster create success is using flavour with disk = 0MB while the result is compute HDD storage being consumed. So with the above, please may I clarify on the following? 1. It seems that 0MB disk flavours may not be supported with magnum now? Could the experts confirm? :) Is there another way that I should be configuring this so that compute node disk is not being consumed (because it is slow and has limited capacity). 2. The issue #1 looks like a bug to me, is it known? If not, is this mail enough to get it realised? Pastebin links as mentioned [1] http://paste.openstack.org/show/797316/ [2] http://paste.openstack.org/show/797318/ Many thanks, Regards, Tony Pearce

2 4

[ptg][glance] Wallaby PTG planning etherpad
by Abhishek Kekane 02 Sep '20

02 Sep '20

Hi All, The Wallaby PTG will be held next month from Oct 26 to Oct 30. I have booked some time slots in the ethercalc[0], however these are not fixed, if the booked time does not suit your time, please let me know. I have also created a planning etherpad [1] where you can add your topics for discussion during PTG. [0]https://ethercalc.openstack.org/7xp2pcbh1ncb. [1] https://etherpad.opendev.org/p/Glance-Wallaby-PTG-planning Thank you, Abhishek Kekane

1 0

Issue with heat and magnum
by Alexander Dibbo - UKRI STFC 02 Sep '20

02 Sep '20

Hi, I am having an issue with magnum creating clusters when I have multiple active heat-engine daemons running. I get the following error in the heat engine logs: 2020-08-14 10:36:30.237 598383 INFO heat.engine.resource [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] signal SoftwareDeployment "master_config_deployment" [67ba9ce2-aba5-4c15-a7ea -6b774659a0e2] Stack "kubernetes-test-26-3uzjqqob47fh-kube_masters-mhctjio2b4gh-0-pbhumflm5mn5" [dc66e4d9-0c9b-4b18-a2c6-dd9724fa51a9] : Authentication cannot be scoped to multiple target s. Pick one of: project, domain, trust or unscoped 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource Traceback (most recent call last): 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2462, in _handle_signal 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource signal_result = self.handle_signal(details) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/engine/resources/openstack/heat/software_deployment.py", line 514, in handle_signal 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource timeutils.utcnow().isoformat()) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 788, in signal_software_deployment 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource version='1.6') 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/rpc/client.py", line 89, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return client.call(ctxt, method, **kwargs) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 165, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource msg_ctxt = self.serializer.serialize_context(ctxt) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/messaging.py", line 46, in serialize_context 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource _context = ctxt.to_dict() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 185, in to_dict 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource 'roles': self.roles, 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 315, in roles 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource self._load_keystone_data() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 292, in wrapped_f 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self.call(f, *args, **kw) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 358, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource do = self.iter(retry_state=retry_state) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 319, in iter 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return fut.result() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/concurrent/futures/_base.py", line 422, in result 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self.__get_result() 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/tenacity/__init__.py", line 361, in call 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource result = fn(*args, **kwargs) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 306, in _load_keystone_data 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource auth_ref = self.auth_plugin.get_access(self.keystone_session) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource self.auth_ref = self.get_auth_ref(session) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/generic/base.py", line 208, in get_auth_ref 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource return self._plugin.get_auth_ref(session, **kwargs) 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource File "/usr/lib/python2.7/site-packages/keystoneauth1/identity/v3/base.py", line 144, in get_auth_ref 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource message='Authentication cannot be scoped to multiple' 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource AuthorizationFailure: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped 2020-08-14 10:36:30.237 598383 ERROR heat.engine.resource 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service [req-a2c862eb-370c-4e91-a2c6-dca32c7872ce - - - - -] Unhandled error in asynchronous task: ResourceFailure: AuthorizationFailure: resources.master_config_deployment: Authentication cannot be scoped to multiple targets. Pick one of: project, domain, trust or unscoped 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service Traceback (most recent call last): 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 132, in log_exceptions 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service gt.wait() 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 181, in wait 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service return self._exit_event.wait() 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/event.py", line 132, in wait 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service current.throw(*self._exc) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/eventlet/greenthread.py", line 221, in main 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service result = function(*args, **kwargs) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 123, in _start_with_trace 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service return func(*args, **kwargs) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1871, in _resource_signal 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service needs_metadata_updates = rsrc.signal(details, need_check) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2500, in signal 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service self._handle_signal(details) 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service File "/usr/lib/python2.7/site-packages/heat/engine/resource.py", line 2480, in _handle_signal 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service raise failure 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service ResourceFailure: AuthorizationFailure: resources.master_config_deployment: Authentication cannot be scoped to multiple targets. Pi ck one of: project, domain, trust or unscoped 2020-08-14 10:36:30.890 598383 ERROR heat.engine.service Each of the individual heat-engine daemons create magnum clusters correctly when they are the only ones online. Attached are the heat and magnum config files. Any ideas where to look would be appreciated? Regards Alexander Dibbo - Cloud Architect / Cloud Operations Group Leader For STFC Cloud Documentation visit https://stfc-cloud-docs.readthedocs.io<https://stfc-cloud-docs.readthedocs.io/> To raise a support ticket with the cloud team please email cloud-support(a)gridpp.rl.ac.uk<mailto:cloud-support@gridpp.rl.ac.uk> To receive notifications about the service please subscribe to our mailing list at: https://www.jiscmail.ac.uk/cgi-bin/webadmin?A0=STFC-CLOUD To receive fast notifications or to discuss usage of the cloud please join our Slack: https://stfc-cloud.slack.com/ This email and any attachments are intended solely for the use of the named recipients. If you are not the intended recipient you must not use, disclose, copy or distribute this email or any of its attachments and should notify the sender immediately and delete this email from your system. UK Research and Innovation (UKRI) has taken every reasonable precaution to minimise risk of this email or any attachments containing viruses or malware but the recipient should carry out its own virus and malware checks before opening the attachments. UKRI does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKRI business are solely those of the author and do not represent the views of UKRI.

2 1

[ptg][cyborg] Cyborg Wallaby PTG schedule and resources
by yumeng bao 02 Sep '20

02 Sep '20

Hi team, The next virtual Wallaby PTG will be held next month from Oct 26 to Oct 30. I have booked the same time slots in the ethercalc[0] as our last PTG did. However, these are not fixed, if the booked time does not suit your time, Please don't hesitate to add your availability to the doodle[1]. BTW, PTG Topics are open to collect and you can add to the etherpad[2]. [0]https://ethercalc.openstack.org/7xp2pcbh1ncb. [1]https://doodle.com/poll/mrudbrn87vh48ayq [2]https://etherpad.opendev.org/p/cyborg-wallaby-goals Regards， Yumeng

1 0

[scientific-sig] SIG Meeting: Ironic kexec, CANOPIE-HPC and the PTG
by Stig Telfer 02 Sep '20

02 Sep '20

Hi All - We have a Scientific SIG meeting on IRC channel #openstack-meeting at 2100 UTC (about 45 minutes). Everyone is welcome. Agenda is here: https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_September_1st_20… Today we'd like to talk about supporting the development of kexec support for Ironic deployments. Also some up-coming CFPs and PTG participation. Cheers, Stig

1 0

[nova] providing a local disk to a Nova instance
by Thomas Goirand 02 Sep '20

02 Sep '20

Hi Nova team! tl;dr: we would like to contribute giving instances access to physical block devices directly on the compute hosts. Would this be accepted? Longer version: About 3 or 4 years ago, someone wrote a spec, so we'd be able to provide a local disk of a compute, directly to a VM to use. This was then rejected, because at the time, Cinder had the blockdevice drive, which was more or less achieving the same thing. Unfortunately, because nobody was maintaining the blockdevice driver in Cinder, and because there was no CI that could test it, the driver got removed. We've investigated how we could otherwise implement it, and one solution would be to use Cinder, but then we'd be going through an iSCSI export, which would drastically reduce performances. Another solution would be to manage KVM instances by hand, not touching anything to libvirt and/or OpenVSwitch, but then we would loose the ease of using the Nova API, so we would prefer to avoid this direction. So we (ie: employees in my company) need to ask the Nova team: would you consider a spec to do what was rejected before, since there's now no other good enough alternative? Our current goal is to be able to provide a disk directly to a VM, so that we could build Ceph clusters with an hyper-converged model (ie: storage hosted on the compute nodes). In this model, we wouldn't need live-migration of a VM with an attached physical block device (though the feature could be added on a later stage). Before we start investigating how this can be done, I need to know if this has at least some chances to be accepted or not. If there is, then we'll probably start an experimental patch locally, then write a spec to properly start this project. So please let us know. Cheers, Thomas Goirand (zigo)

4 3

[neutron] Bug deputy report August 24 - 30
by Rodolfo Alonso Hernandez 02 Sep '20

02 Sep '20

Hello: This is the list of bugs in the period from August 24 to August 30. Critical: (none) High: - https://bugs.launchpad.net/neutron/+bug/1892861: [neutron-tempest-plugin] If paramiko SSH client connection fails because of authentication, cannot reconnect (unassigned) - https://bugs.launchpad.net/neutron/+bug/1893314: set neutron quota not take effect (ZhouHeng, https://review.opendev.org/#/c/748578/) Medium: - https://bugs.launchpad.net/neutron/+bug/1892496: 500 on SG deletion: Cannot delete or update a parent row (unassigned) - https://bugs.launchpad.net/neutron/+bug/1888878: Data not present in OVN IDL when requested (fix released) - https://bugs.launchpad.net/neutron/+bug/1892846: AttributeError in l3-agent changing HA router to non-HA (Slawek, https://review.opendev.org/#/c/747922/) Low: - https://bugs.launchpad.net/neutron/+bug/1892741: neutron-sriov-nic-agent cannot be disabled (unassigned) - https://bugs.launchpad.net/neutron/+bug/1893188: [tempest] Randomize subnet CIDR to avoid test clashes (Lajos, https://review.opendev.org/#/c/749041/) Wishlist: - https://bugs.launchpad.net/neutron/+bug/1893625: Implement router gateway IP QoS in OVN backend (zhanghao, https://review.opendev.org/#/c/749012/) Incomplete: - https://bugs.launchpad.net/neutron/+bug/1893015: ping with large package size fails Regards.

3 2

2020 Virtual Summit: Forum Submissions Now Accepted
by Jimmy McArthur 01 Sep '20

01 Sep '20

Hello Everyone! We are now accepting Forum [1] submissions for the 2020 Virtual Open Infrastructure Summit [2]. Please submit your ideas through the Summit CFP tool [3] through September 14th. Don't forget to put your brainstorming etherpad up on the Shanghai Forum page [4]. This is not a classic conference track with speakers and presentations. OSF community members (participants in development teams, operators, working groups, SIGs, and other interested individuals) discuss the topics they want to cover and get alignment on and we welcome your participation. The Forum is your opportunity to help shape the development of future project releases. More information about the Forum [1]. The timeline for submissions is as follows: Aug 31st | Formal topic submission tool opens: https://cfp.openstack.org. Sep 14th | Deadline for proposing Forum topics. Scheduling committee meeting to make draft agenda. Sep 21st | Draft Forum schedule published. Crowd sourced session conflict detection. Forum promotion begins. Sept 28th | Forum schedule final Oct 19th | Forum begins! If you have questions or concerns, please reach out to speakersupport(a)openstack.org. Cheers, Jimmy [1] https://wiki.openstack.org/wiki/Forum [2] https://www.openstack.org/summit/2020/ [3] https://cfp.openstack.org [4]https://wiki.openstack.org/wiki/Forum/Virtual2020

1 1

[tripleo][ussuri][undercloud][tripleo_rabbitmq] do not start after reboot
by Ruslanas Gžibovskis 01 Sep '20

01 Sep '20

Hi all, I think this issue might be known to RHOSP16 [1], but it is same in RDO - ussuri with centos8. if anything is needed, I could try to help/test. my workaround is to run openstack undercloud install --force-stack-update [1] https://access.redhat.com/solutions/5269241 -- Ruslanas Gžibovskis +370 6030 7030

1 0

[horizon] horizon-integration-tests job is broken
by Ivan Kolodyazhny 01 Sep '20

01 Sep '20

Hi team, Please, do not trigger recheck if horizon-integration-tests failed like [1]: ______________ TestDashboardHelp.test_dashboard_help_redirection _______________ 'NoneType' object is not iterable While I'm trying to figure out what is happening there [2], any help with troubleshooting is welcome. [1] https://51bb980dc10c72928109-9873e0e5415ff38d9f1a5cc3b1681b19.ssl.cf1.rackc… [2] https://review.opendev.org/#/c/749011 Regards, Ivan Kolodyazhny, http://blog.e0ne.info/

1 1

[ops] Restructuring OSOPS tools
by Sean McGinnis 01 Sep '20

01 Sep '20

Hello everyone, We recently expanded the scope of the Ops Docs SIG to also include any ops tooling. I think it's now time to move on to the next step of actually getting some of the old tooling in place and organized how we want it. We have several semi-abandoned repos from back when there was more work being done on ops tooling. During the great rebranding, those all were moved under the x/ namespace: https://opendev.org/x/?tab=&sort=recentupdate&q=osops Since these are now owned by an official SIG, we can move this content back under the openstack/ namespace. That should help increase visibility somewhat, and make things look a little more official. It will also allow contributors to tooling to get recognition for contributing to an import part of the OpenStack ecosystem. I do think it's can be a little more difficult to find things spread out over several repos though. For simplicity with finding tooling, as well as watching for reviews and helping with overall maintenance, I would like to move all of these under a common openstack/osops. Under that repo, we can then have a folder structure with tools/logging, tools/monitoring, etc. Then with everything in one place, we can have docs published in one place that helps find everything and easily links between tools. We can also capture some metadata about the tools, and use that to reflect their state in those docs. Please let me know if there are any objects to this plan. Otherwise, I will start cleaning things up and getting it staged in a new repo to be imported as an official repo owned by the SIG. Thanks! Sean

3 2

[TripleO][Ussuri] image prepare, cannot download containers image prepare recently
by Ruslanas Gžibovskis 01 Sep '20

01 Sep '20

Hi all, I have noticed, that recently my undercloud is not able to download images [0]. I have provided newly generated containers-prepare-parameter.yaml and outputs from container image prepare providing --verbose and later beginning of --debug (in the end) [0] were there any changes? As "openstack tripleo container image prepare default --output-env-file containers-prepare-parameter.yaml --local-push-destination" have prepared a bit different file, compared what was previously: NEW # namespace: docker.io/tripleou VS namespace: docker.io/tripleomaster # OLD [0] - http://paste.openstack.org/show/rBCNAQJBEe9y7CKyi9aG/ -- Ruslanas Gžibovskis +370 6030 7030

2 1