January 2024 - openstack-discuss - lists.openstack.org

[magnum] k8s-keystone-auth pod CrashLoopBackOff
by Eugen Block 16 Jan '24

16 Jan '24

Hi there, I've noticed several magnum related threads and thought I'd give it a try while I keep digging. I've only recently started to work with magnum, and this is how far I get. Note that this openstack is still running on version Victoria, I'm working towards migrating it to a newer release. My coe cluster template currently looks like this (I'll omit the defaults): | labels | {'kube_tag': 'v1.18.2', 'hyperkube_prefix': 'k8s.gcr.io/'} | | https_proxy | http://<IP>:<PORT> | | http_proxy | http://<IP>:<PORT> | | image_id | fedora-coreos33 | | dns_nameserver | <MY_DNS> | I've already tried quite a lot of different kube_tags and hyperkube_prefixes, this combination gets me at least to have some running containers: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7fbe3f8f308f docker.io/openstackmagnum/heat-container-agent:victoria-stable-1 /usr/bin/start-he... 42 minutes ago Up 42 minutes ago heat-container-agent a8b6ae357034 k8s.gcr.io/hyperkube:v1.18.2 kube-apiserver --... 40 minutes ago Up 40 minutes ago kube-apiserver cb7774fbf849 k8s.gcr.io/hyperkube:v1.18.2 kube-controller-m... 40 minutes ago Up 40 minutes ago kube-controller-manager 448f1c02cbd4 k8s.gcr.io/hyperkube:v1.18.2 kube-scheduler --... 40 minutes ago Up 40 minutes ago kube-scheduler 4be28a0a2271 k8s.gcr.io/hyperkube:v1.18.2 kubelet --logtost... 40 minutes ago Up 40 minutes ago kubelet 542ffdb081e7 k8s.gcr.io/hyperkube:v1.18.2 kube-proxy --logt... 40 minutes ago Up 40 minutes ago kube-proxy 61874a4e4595 quay.io/coreos/etcd:v3.4.6 /usr/local/bin/et... 40 minutes ago Up 40 minutes ago etcd But not all pods are starting: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl get pods -o wide --kubeconfig=/etc/kubernetes/admin.conf -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-786ffb7797-dpkbx 1/1 Running 0 25m 10.100.0.3 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system coredns-786ffb7797-mslfk 1/1 Running 0 25m 10.100.0.2 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system dashboard-metrics-scraper-6b4884c9d5-nrvbg 1/1 Running 0 25m 10.100.0.5 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system k8s-keystone-auth-qt7n5 0/1 CrashLoopBackOff 4 5m12s 10.0.0.116 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system kube-dns-autoscaler-75859754fd-69zxx 0/1 Pending 0 25m <none> <none> <none> <none> kube-system kube-flannel-ds-cn5jf 1/1 Running 0 25m 10.0.0.116 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system kubernetes-dashboard-c98496485-qw4hl 1/1 Running 0 25m 10.100.0.4 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system magnum-metrics-server-79556d6999-hkqcw 0/1 Pending 0 25m <none> <none> <none> <none> I didn't look too deep into the two other failing pods, I think the keystone pod is more important at this point or maybe even is required for the other pods to start, I don't know yet. I can pull the image manually and the "describe pod" command also shows that the container was pulled successfully, but I can't find other hints what is actually failing. Here is the full output of "describe pod": ---snip--- [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system describe pod/k8s-keystone-auth-qt7n5 Name: k8s-keystone-auth-qt7n5 Namespace: kube-system Priority: 0 Node: kubernetes-cluster20-buuxqog6cfnn-master-0/10.0.0.116 Start Time: Mon, 15 Jan 2024 09:13:16 +0000 Labels: controller-revision-hash=6875f96b46 k8s-app=k8s-keystone-auth pod-template-generation=1 Annotations: kubernetes.io/psp: magnum.privileged Status: Running IP: 10.0.0.116 IPs: IP: 10.0.0.116 Controlled By: DaemonSet/k8s-keystone-auth Containers: k8s-keystone-auth: Container ID: docker://adca0b9a9eb011f22530886a42bf862b4cfa7e6b7965cd77d6dac4f37222b827 Image: docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0 Image ID: docker-pullable://k8scloudprovider/k8s-keystone-auth@sha256:ab72a7e5b9eca9af2762796690be800bce2de25dee8bd5ad55d0996b29c85146 Port: 8443/TCP Host Port: 8443/TCP Args: ./bin/k8s-keystone-auth --tls-cert-file /etc/kubernetes/certs/server.crt --tls-private-key-file /etc/kubernetes/certs/server.key --policy-configmap-name k8s-keystone-auth-policy --keystone-url http://<CONTROLLER>:5000/v3 --sync-configmap-name keystone-sync-policy --keystone-ca-file /etc/kubernetes/ca-bundle.crt --listen 127.0.0.1:8443 State: Terminated Reason: Error Exit Code: 1 Started: Mon, 15 Jan 2024 09:23:52 +0000 Finished: Mon, 15 Jan 2024 09:23:52 +0000 Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 15 Jan 2024 09:18:41 +0000 Finished: Mon, 15 Jan 2024 09:18:41 +0000 Ready: False Restart Count: 6 Requests: cpu: 200m Environment: <none> Mounts: /etc/kubernetes from ca-certs (ro) /etc/kubernetes/certs from k8s-certs (ro) /var/run/secrets/kubernetes.io/serviceaccount from k8s-keystone-auth-token-vhm4r (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: k8s-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/certs HostPathType: DirectoryOrCreate ca-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes HostPathType: DirectoryOrCreate k8s-keystone-auth-token-vhm4r: Type: Secret (a volume populated by a Secret) SecretName: k8s-keystone-auth-token-vhm4r Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: :NoSchedule :NoExecute CriticalAddonsOnly node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/network-unavailable:NoSchedule node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/pid-pressure:NoSchedule node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unschedulable:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/k8s-keystone-auth-qt7n5 to kubernetes-cluster20-buuxqog6cfnn-master-0 Warning Failed 9m52s kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Error: ErrImagePull Warning Failed 9m52s kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Failed to pull image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0": rpc error: code = Unknown desc = Get "https://registry-1.docker.io/v2/k8scloudprovider/k8s-keystone-auth/manifest…": Internal Server Error Normal Pulling 9m37s (x5 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Pulling image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0" Normal Pulled 9m36s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Successfully pulled image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0" Normal Created 9m35s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Created container k8s-keystone-auth Normal Started 9m35s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Started container k8s-keystone-auth Warning BackOff 33s (x48 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Back-off restarting failed container ---snip--- I'm obviously not allowed to get logs from the pod: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system logs pod/k8s-keystone-auth-qt7n5 Error from server: Get https://10.0.0.116:10250/containerLogs/kube-system/k8s-keystone-auth-qt7n5/…: Forbidden I appreciate any hints! Thanks! Eugen

1 3

[keystone] Keystone as a Service Provider, 401 - Unauthorized Exception for SSO
by joeldag7＠gmail.com 16 Jan '24

16 Jan '24

We're currently working on implementing SSO for our university's OpenStack environment using OpenStack Ansible (Yoga version). Our goal is to allow students to log in with their university credentials via an Identity Provider - currently we're using our own Keycloak instance as IdP. The plan is to automatically create a new project for first time users, while returning students will be redirected to their existing projects to manage their instances. For initial testing, we're directing all users to a single project. Eventually, we would like to script the dynamic creation of projects on the controller node for first-time logins later. However, we've run into some problems: Our Keystone configuration in the user_variables.yml file is as follows: ``` keystone_sp: apache_mod: 'mod_auth_openidc' cadf_notifications: true trusted_dashboard_list: - "https://openstack-example.com/auth/websso/" - "http://example.keycloak.com/realms/openstack*" - "https://openstack-example.com/auth/login/" trusted_idp_list: - name: "oidc-idp" oidc_provider_metadata_url: http://example.keycloak.com/realms/openstack/.well-known/openid-configurati… oidc_client_id: ******** oidc_client_secret: ********* oidc_crypto_passphrase: ****** oidc_redirect_uri: https://openstack-example.com/v3/OS-FEDERATION/identity_providers/oidc-idp/… entity_ids: - 'http://exmaple.keycloak.com/realms/openstack' federated_identities: - domain: default project: fedproject group: fedgroup role: member protocols: - name: openid mapping: name: openid-mapping rules: - remote: - type: email local: - group: name: fedgroup domain: name: Default user: name: '{0}' ``` This setup is based on the openstack documentation (https://docs.openstack.org/openstack-ansible-os_keystone/yoga/configure-fed…) After many attempts with different configurations, this one successfully redirects to our Keycloak IdP for authentication. However, after login, when redirected back to OpenStack, we get a 401 Unauthorised error stating "The request you have made requires authentication". The SAML tracer logs confirm that Keycloak issues a valid token along with the user information. Despite this, we find no additional error details in the Keystone container, Apache2 and controller logs. This leads me to believe that the problem may lie in the mapping configuration, although it appears to work when tested with the keystone-manage mapping-engine tool. But still, the doc (https://docs.openstack.org/keystone/queens/advanced-topics/federation/mappi…) claims that a 401 exception could be caused by the mapping. Any insight or suggestions on how to solve this problem would be greatly appreciated, particularly regarding the mapping process during authentication. I'm also looking for advice on how to extract more detailed error information. At the moment, the logs don't reveal much beyond the 401 error message. Am I missing something or is this all the information OpenStack typically provides in such cases? Thank you for your help and guidance.

1 0

OpenStack D Release Name
by Allison Price 15 Jan '24

15 Jan '24

Hi everyone, I am excited to share that the name for the OpenStack D release, scheduled for 2H 2024 is Dalmatian. (Congrats, Amy!) We look forward to developing the 30th release of OpenStack! Cheers, Allison

4 3

[tc] Next Technical Committee Meeting Jan 16 2024, 1800 UTC
by Jay Faulkner 15 Jan '24

15 Jan '24

Hi all, The next weekly OpenStack Technical Committee meeting is tomorrow, Tuesday Jan 16 at 1800 UTC in #openstack-tc on OFTC. The proposed agenda is: Agenda - Roll call - Follow up on tracked action items - JayF - Email mailing list a summary of the discussion around Monasca, highlighting issues around trust and project activity along with a link to the meeting log. - slaweq - Reach out privately to volunteer to maintain Monasca about official vs unofficial options - gmann - Reach out privately to volunteer to maintain Senlin about next steps - Gate health check - Implementation of Unmaintained branch statuses - Inactive projects for Caracal - It is past C-2, these projects will not be in Caracal release. Would we like to take further action in Caracal? - Monasca - Senlin - Freezer - Solum - EC2 API - Sahara - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Open Discussion and Reviews Thanks, Jay Faulkner

1 0

[neutron] Bug deputy - week of Jan 8
by Sławek Kapłoński 15 Jan '24

15 Jan '24

Hi, It was pretty quiet week. Below is summary of the bugs opened this last week and its current status. ## High * https://bugs.launchpad.net/neutron/+bug/2048745 - [ovn] FIP not working when mixing vlan and geneve tenant networks - not assigned, many things to consider before we will start working on this improvement really * https://bugs.launchpad.net/neutron/+bug/2048785 - Trunk parent port (tpt port) vlan_mode is wrong in ovs - WIP fix already started by Bence: https://review.opendev.org/c/openstack/neutron/+/905125 * https://bugs.launchpad.net/neutron/+bug/2049265 - [OVN] The ``MaintenanceWorker`` shoudl register the "Private_Chassis" table in the SB connection - assigned to Terry Wilson, fixes already proposed https://review.opendev.org/q/I711996b7644e80bc195833e4429e4d745728f9cf ## Medium * https://bugs.launchpad.net/neutron/+bug/2048979 - [ml2][ovs] ports without local vlan tag are processed on openflow security group - fix proposed by LIU: https://review.opendev.org/c/openstack/neutron/+/905289 ## Others * https://bugs.launchpad.net/neutron/+bug/2048773 - neutron-meter-agent do not support ovn driver - unassigned, it isn't really completly new RFE IMO but probably approach to take to close this gap may be discussed e.g. in the drivers meeting, * https://bugs.launchpad.net/neutron/+bug/2049398 - [Tempest] neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_delete_policies_while_tenant_attached_to_net "Created Network is Not an External Network" Error - needs some more info to be triaged, I will take care of it -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[Image Encryptione] Cancelling tomorrows meeting
by Josephine Seifert 14 Jan '24

14 Jan '24

Hi, unfortunately I have to cancel tomorrows meeting. greetings, Luzi (Josephine)

1 0

[kolla]Boot one VM with two GPU(in same numa)by pci passthrough cannot have GPUDirect P2P capability
by dazhaoyu＠126.com 13 Jan '24

13 Jan '24

I have two GPU cards, all of them was connect with one same numa CPU socket as below link info: https://paste.opendev.org/show/b7Qi8qCnbLVxO2W0JdQw/ I can boot one nova instance successfully with the two GPU cards by PCI Passthrough way. but in the booted instances, use deviceQuery method would get the below message: Peer access from NVIDIA RTX 6000(GPU0) -> NVIDIA RTX 6000(GPU1): NO Peer access from NVIDIA RTX 6000(GPU1) -> NVIDIA RTX 6000(GPU0): NO The expected return should be as below： Peer access from NVIDIA RTX 6000(GPU0) -> NVIDIA RTX 6000(GPU1): YES Peer access from NVIDIA RTX 6000(GPU1) -> NVIDIA RTX 6000(GPU0): YES so that the memory can be shared between the two GPUs. I'm running Openstack Xena release in Intel Xeon Gold 5220R CPU

1 0

[release] Release countdown for week R-11, Jan 15-19
by Elõd Illés 12 Jan '24

12 Jan '24

Development Focus ----------------- We are now past the Caracal-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 22nd, 2024). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 29th, 2024) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 29th, 2024. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 14th, 2024) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 28th, 2024) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: February 22nd, 2024 (R-6 week) Client library freeze: February 29th, 2024 (R-5 week) Caracal-3 milestone: February 29th, 2024 (R-5 week) 2024.1 Caracal final release: April 3rd, 2024 2024.2 'D' virtual PTG ( https://openinfra.dev/ptg/ ): April 8th-12th, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

[openstack][keystone] user who mapped from openid cannot create application credentials
by Nguyễn Hữu Khôi 12 Jan '24

12 Jan '24

Hello. I have a problem that I use openid for authentication to my cloud but my users cannot create application credentials. I must add them to a group via Horizon UI to use application credentials. Is there any way to do that without adding users to a group from Horizon? Thank you. Nguyen Huu Khoi

1 1

[kolla-ansible][magnum]Private CA in Fedora-CoreOS
by Franck VEDEL 12 Jan '24

12 Jan '24

Hello, I have two platforms: on the first (Yoga), which is connected with public names and addresses, certificates issued by trusted CAs, no problem, creating a cluster template and deploying a cluster based on Fedora-CoreOS-35 works perfectly. On the second, I have a private certification authority (and privates addresses) and when I create a template then deploy a cluster, obviously there is a problem (in each node of the cluster) with the certificate issued by an untrusted CA. For the cluster to be deployed, I need to connect to the nodes, copy my CA certificate to /etc/pki/ca-trust/source/anchors/ then run the "update-ca-trust" command. My question is: is it possible to deploy a cluster using this method but without having to do what I am obliged to do if the CA is not trusted? If the answer exists, I couldn't find it in the documentation, I apologize. Thanks in advance Franck

1 0