July 2023 - openstack-discuss - lists.openstack.org

[kolla-ansible][octavia]
by Nguyễn Hữu Khôi 15 Jul '23

15 Jul '23

Hello guys. I am using openstack yoga by kolla-ansible but when i reconfigure octavia then it stucked at TASK [octavia : Create loadbalancer management network] \"Unable to create the network. The VLAN xxx on physical network physnet1 is in use.\ I see that it cannot create lb network because someone used it. Will we need to change some kolla-ansible tasks to fix it? Thank you very much. Nguyen Huu Khoi

2 3

[neutron] Neutron drivers meeting
by Rodolfo Alonso Hernandez 14 Jul '23

14 Jul '23

Hello Neutrinos: Today's meeting will be cancelled due to the lack of quorum. At least three of our drivers members [1] are in PTO today, thus we'll postpone the requested RFE discussion [2] to next week. Sorry for the inconveniences. Regards. [1]https://launchpad.net/~neutron-drivers/+members [2]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[cinder] I/O throttling using blkio cgroups
by Jon Bernard 14 Jul '23

14 Jul '23

I started to look at a cinder bug [1] to migrate our cgroup usage from v1 to v2 and I'm wondering: is anyone using this feature to throttle cinder's local I/O per cinder-volume instance? Docker since 20.10 and kubernetes since version 1.19 support cgroup v2. Since cinder services are deployed within containers, I think it makes sense to consider implementing I/O throttling at the container-level as opposed to application-specific code within a particular service (cinder in this case, but I believe glance also leverages cgroups). That would mean instead of updating our implementation to use the v2 interface, we could drop our cgroup code entirely in favor of a more generic solution that could be applied to any container that performs local I/O. Would anyone be opposed to this idea? Thoughts? 1: https://bugs.launchpad.net/cinder/+bug/1942203 -- Jon

1 0

SNAT failure with OVN under Antelope
by Gary Molenkamp 13 Jul '23

13 Jul '23

Good morning, I'm having a problem with snat routing under OVN but I'm not sure if something is mis-configured or just my understanding of how OVN is architected is wrong. I've built a Zed cloud, since upgraded to Antelope, using the Neutron Manual install method here: https://docs.openstack.org/neutron/latest/install/ovn/manual_install.html I'm using a multi-tenent configuration using geneve and the flat provider network is present on each hypervisor. Each hypervisor is connected to the physical provider network, along with the tenent network and is tagged as an external chassis under OVN. br-int exists, as does br-provider ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw For most cases, distributed FIP based connectivity is working without issue, but I'm having an issue where VMs without a FIP are not always able to use the SNAT services of the tenent network router. Scenario: Internal network named cs3319: with subnet 172.31.100.0/23 Has a router named cs3319_router with external gateway set (snat enabled) This network has 3 vms: - #1 has a FIP and can be accessed externally - #2 has no FIP, can be accessed via VM1 and can access external resources via SNAT (ie OS repos, DNS, etc) - #3 has no FIP, can be accessed via VM1 but has no external SNAT connectivity From what I can tell, the chassis config is correct, compute05 is the hypervisor and the faulty VM has a port binding on this hypervisor: ovn-sbctl show ... Chassis "8e0fa17c-e480-4b60-9015-bd8833412561" hostname: compute05.cloud.sci.uwo.ca Encap geneve ip: "192.168.0.105" options: {csum="true"} Port_Binding "7a5257eb-caea-45bf-b48c-620c5dff4b39" Port_Binding "50e16602-78e6-429b-8c2f-e7e838ece1b4" Port_Binding "f121c9f4-c3fe-4ea9-b754-a809be95a3fd" The router has the candidate gateways, and the snat set: ovn-nbctl show 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c router 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c (neutron-389439b5-07f8-44b6-a35b-c76651b48be5) (aka cs3319_public_router) port lrp-44ae1753-845e-4822-9e3d-a41e0469e257 mac: "fa:16:3e:9a:db:d8" networks: ["129.100.21.94/22"] gateway chassis: [5c039d38-70b2-4ee6-9df1-596f82c68106 99facd23-ad17-4b68-a8c2-1ff6da15ac5f 1694116c-6d30-4c31-b5ea-0f411878316e 2a4bbaf9-228a-462e-8970-0cdbf59086e6 9332c61b-93e1-4a70-9547-701a014bfd98] port lrp-509bba37-fa06-42d6-9210-2342045490db mac: "fa:16:3e:ff:0f:3b" networks: ["172.31.100.1/23"] nat 11e0565a-4695-4f67-b4ee-101f1b1b9a4f external ip: "129.100.21.94" logical ip: "172.31.100.0/23" type: "snat" nat 21e4be02-d81c-46e8-8fa8-3f94edb4aed1 external ip: "129.100.21.87" logical ip: "172.31.100.49" type: "dnat_and_snat" Each network agent on the hypervisors shows the ovn controller up : OVN Controller Gateway agent | compute05.cloud.sci.uwo.ca | | :-) | UP | ovn-controller The ovs vswitch on the hypervisor looks correct afaict and ovn ports bfd status are all forwarding to other hypervisors. ie: Port ovn-2a4bba-0 Interface ovn-2a4bba-0 type: geneve options: {csum="true", key=flow, remote_ip="192.168.0.106"} bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up} Any advice on where to look would be appreciated. PS. Version info: Neutron 22.0.0-1 OVN 22.12 neutron options: enable_distributed_floating_ip = true ovn_l3_scheduler = leastloaded Thanks Gary -- Gary Molenkamp Science Technology Services Systems/Cloud Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 14

OpenStack "C" Release Voting
by Kristin Barrientos 13 Jul '23

13 Jul '23

Hi everyone, It’s time to vote for the OpenStack “C” Release! The nominees are: - Celine - Caracal - Cashew Get your votes in! Vote here: https://civs1.civs.us/cgi-bin/vote.pl?id=E_852cf3e53467d72e&akey=1504cb6ad9… Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[cinder]cinder qos question.
by Nguyễn Hữu Khôi 13 Jul '23

13 Jul '23

Hello guys. I am trying to use cinder qos. But i have question. - I added qos after a volume was create and I see that qos did not apply to this volume. - i create new volume with assosiate qos then it is ok but when i changed extra specs in qos, volume keep old specs. Is there any way to make qos apply with two above cases. Many thanks.

2 5

Re: [kuryr][tacker] fails to bump the k8s version to v1.26
by Michał Dulko 13 Jul '23

13 Jul '23

On Thu, 2023-07-13 at 10:16 +0000, Ayumu Ueha (Fujitsu) wrote: > Hi Michał, > > Thanks for your reply, > > I trying it in my local machine and check logs, but I am not familiar > with troubleshooting with kubelet/kubeadm, so I do not know why it > errored as for now. > We will continue to investigate, but it would be helpful if you could > share any small findings. > > For your reference, I will attach a log of the kubelet.service when > it failed in my local environment. Looks like kube-apiserver and kube-scheduler are failing to start. Checking logs of them would be my next idea. You might want to use `crictl` to find them. > Thank you. > > Best Regards, > Ueha > > -----Original Message----- > From: Michał Dulko <mdulko(a)redhat.com> > Sent: Thursday, July 13, 2023 3:48 PM > To: openstack-discuss(a)lists.openstack.org > Subject: Re: [kuryr][tacker] fails to bump the k8s version to v1.26 > > On Tue, 2023-07-04 at 07:53 +0000, Ayumu Ueha (Fujitsu) wrote: > > > > > > > > Hi kuryr-kubetenetes team, > > > > Tacker uses the kuryr-kubernetes as the setup for the kubernetes > > Environment. > > In the Bobcat version, we will bump the supported version of > > kubenetes > > to 1.26.6, and when I tried to bump the version in devstack's > > local.conf by the patch [1], the following error occurred. (please > > refer the details to Zuul log > > [2]) > > ============================== > > [wait-control-plane] Waiting for the kubelet to boot up the control > > plane as static Pods from directory "/etc/kubernetes/manifests". > > This > > can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. > > > > Unfortunately, an error has occurred: > > timed out waiting for the condition > > > > This error is likely caused by: > > - The kubelet is not running > > - The kubelet is unhealthy due to a misconfiguration > > of > > the node in some way (required cgroups disabled) > > > > If you are on a systemd-powered system, you can try to troubleshoot > > the error with the following commands: > > - 'systemctl status kubelet' > > - 'journalctl -xeu kubelet' > > > > Additionally, a control plane component may have crashed or exited > > when started by the container runtime. > > To troubleshoot, list all containers using your preferred container > > runtimes CLI. > > Here is one example how you may list all running Kubernetes > > containers > > by using crictl: > > - 'crictl --runtime-endpoint > > unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' > > Once you have found the failing container, you can > > inspect its logs with: > > - 'crictl --runtime-endpoint > > unix:///var/run/crio/crio.sock logs CONTAINERID' > > error execution phase wait-control-plane: couldn't initialize a > > Kubernetes cluster ============================== > > > > I know it is not yet supporting version 1.26 of the K8S at the > > kuryr- > > kubernetes as for now, but do you know any way to avoid the above > > error? > > I suspect this is due to a change from version 1.25, but I'm not > > sure > > which one is affecting... > > > > Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? > > Will the Bobcat release support it? > > Please kindly let me know if you know anything. Thank you. > > Kuryr-kubernetes itself supports K8s 1.26 and 1.27 as we test it with > these versions as part of OpenShift. What doesn't seem to work is the > DevStack plugin, I guess due to some breaking changes in kubelet or > kubeadm. > > I can advise to try to deploy DevStack with these settings on your > local machine and then you should have access to all the logs > required to figure out what is causing the problem with the version > bump. > > > [1] https://review.opendev.org/c/openstack/tacker/+/886935 > > Change the parameters: > > KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 > > CRIO_VERSION: 1.25 to 1.26 > > [2] > > https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219 > > b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816 > > > > Best Regards, > > Ueha > >

1 0

[cinder-backup][kolla] mix cinder-volume and cinder-backup types (nfs, ceph)
by garcetto 13 Jul '23

13 Jul '23

good evening, it is possible to mix cinder-volume backend (say ceph) and cinder-backup backend (say nfs)? and also do incrementals backup? thank you

3 3

[kuryr][tacker] fails to bump the k8s version to v1.26
by Ayumu Ueha (Fujitsu) 13 Jul '23

13 Jul '23

Hi kuryr-kubetenetes team, Tacker uses the kuryr-kubernetes as the setup for the kubernetes Environment. In the Bobcat version, we will bump the supported version of kubenetes to 1.26.6, and when I tried to bump the version in devstack's local.conf by the patch [1], the following error occurred. (please refer the details to Zuul log [2]) ============================== [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster ============================== I know it is not yet supporting version 1.26 of the K8S at the kuryr-kubernetes as for now, but do you know any way to avoid the above error? I suspect this is due to a change from version 1.25, but I'm not sure which one is affecting... Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? Will the Bobcat release support it? Please kindly let me know if you know anything. Thank you. [1] https://review.opendev.org/c/openstack/tacker/+/886935 Change the parameters: KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 CRIO_VERSION: 1.25 to 1.26 [2] https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219b54d9… Best Regards, Ueha

2 2

[horizon][keystone] Adding different rules in the same protocol for federated logon
by James Leong 13 Jul '23

13 Jul '23

Hi all, I have yoga version openstack with the deployment tool of kolla-ansible. I am trying to combine different mapping rules such as allowing user to login to different domain. However, I am not able to do that in a single JSON file. When I try to include different rule in the same JSON file, only the first rule is being considered. Is there a way to allow multiple rule to redirect user to their account in a different domain. Best, James

2 4