January 2024 - openstack-discuss - lists.openstack.org

[neutron] Neutron drivers meeting
by Brian Haley 18 Jan '24

18 Jan '24

Hello Neutrinos, Please remember tomorrow (Friday) we have a Neutron drivers meeting. This is the agenda: https://wiki.openstack.org/wiki/Meetings/NeutronDrivers There are currently two topics: * (lajoskatona): quicky discuss the eventlet hell and what are the first steps: https://review.opendev.org/q/topic:bump-eventlet * (mtomaska): https://bugs.launchpad.net/neutron/+bug/2047849. [RFE] Start using oslo messaging namespaces https://review.opendev.org/c/openstack/neutron/+/905309 I realize there might be a couple of people out, if we do not have quorum will just postpone until next week. Thanks, -Brian

1 0

Problem with failing container-puppet-glance_api_internal during wallaby overcloud deploy.
by davorin.mikulic＠gmail.com 18 Jan '24

18 Jan '24

Hi, I have successfully deployed networks, vips, overcloud nodes and ceph cluster. Now I am proceeding with openstack overcloud deploy command as follows: openstack overcloud deploy --deployed-server --answers-file /home/stack/3XOenvironment/answers.yaml answers.yaml is as follows: (undercloud) [stack@uc4tb27 3XOenvironment]$ cat answers.yaml templates: /home/stack/generated-openstack-tripleo-heat-templates networks: /home/stack/3XOenvironment/40-network_data.yaml roles: /home/stack/3XOenvironment/30-roles_data.yaml environments: - /home/stack/3XOenvironment/deployed-server-environment.yaml - /home/stack/3XOenvironment/network-environment.yaml - /home/stack/3XOenvironment/overcloud-networks-deployed.yaml - /home/stack/3XOenvironment/overcloud-baremetal-deployed.yaml - /home/stack/3XOenvironment/deployed-vips.yaml - /home/stack/3XOenvironment/00-containers-prepare-parameter.yaml - /home/stack/3XOenvironment/10-docker-ha.yaml - /home/stack/3XOenvironment/cephadm.yaml - /home/stack/3XOenvironment/deployed-ceph.yaml We are not using a TLS setup anywhere in the overcloud or the undercloud. During the deploy, after the stack has been successfully created and deploy proceeds to puppet configuration of containers in step 1, I am getting the following error: 2024-01-17 09:30:39.360378 | 000c29b1-9f06-28d0-4abe-00000000d389 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-1 | 0:18:04.150199 | 324.03s 2024-01-17 09:30:43.693815 | | WARNING | ERROR: Can't run container container-puppet-glance_api_internal stderr: + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 08:25:20 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 08:25:20 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 08:25:20 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 08:25:20 puppet-user: (file & line not available) <13>Jan 17 08:25:20 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 08:25:20 puppet-user: (file & line not available) <13>Jan 17 08:25:20 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 08:25:20 puppet-user: has no parameter named 'bind_port' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'log_file' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-2.tb27.cld + rc=1 + '[' false = false ']' + set +x 2024-01-17 09:30:43.695467 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-2 | error={"changed": false, "msg": "Failed containers: container-puppet-glance_api_internal"} 2024-01-17 09:30:43.696471 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-2 | 0:18:08.486292 | 323.73s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=238 changed=96 unreachable=0 failed=0 skipped=135 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=240 changed=96 unreachable=0 failed=0 skipped=130 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=240 changed=96 unreachable=0 failed=0 skipped=130 rescued=0 ignored=1 overcloud-controller-0 : ok=295 changed=140 unreachable=0 failed=1 skipped=175 rescued=0 ignored=1 overcloud-controller-1 : ok=306 changed=140 unreachable=0 failed=1 skipped=164 rescued=0 ignored=1 overcloud-controller-2 : ok=306 changed=140 unreachable=0 failed=1 skipped=164 rescued=0 ignored=1 overcloud-novacompute-0 : ok=281 changed=132 unreachable=0 failed=0 skipped=165 rescued=0 ignored=1 overcloud-novacompute-1 : ok=288 changed=132 unreachable=0 failed=0 skipped=158 rescued=0 ignored=1 overcloud-novacompute-2 : ok=288 changed=132 unreachable=0 failed=0 skipped=158 rescued=0 ignored=1 undercloud : ok=25 changed=9 unreachable=0 failed=0 skipped=2 rescued=0 ignored=1 2024-01-17 09:30:43.764701 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.765549 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1578 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.766211 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:18:08.556038 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.766837 | UUID | Info | Host | Task Name | Run Time 2024-01-17 09:30:43.767508 | 000c29b1-9f06-28d0-4abe-00000000d389 | SUMMARY | overcloud-controller-1 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 324.03s 2024-01-17 09:30:43.768148 | 000c29b1-9f06-28d0-4abe-00000000d33c | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 323.73s 2024-01-17 09:30:43.768828 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | SUMMARY | overcloud-controller-2 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 323.73s 2024-01-17 09:30:43.769558 | 000c29b1-9f06-28d0-4abe-00000000b974 | SUMMARY | overcloud-novacompute-0 | Pre-fetch all the containers | 181.76s 2024-01-17 09:30:43.770215 | 000c29b1-9f06-28d0-4abe-00000000b9f7 | SUMMARY | overcloud-novacompute-1 | Pre-fetch all the containers | 175.41s 2024-01-17 09:30:43.770929 | 000c29b1-9f06-28d0-4abe-00000000ba70 | SUMMARY | overcloud-novacompute-2 | Pre-fetch all the containers | 169.84s 2024-01-17 09:30:43.771586 | 000c29b1-9f06-28d0-4abe-00000000c777 | SUMMARY | overcloud-controller-2 | Wait for puppet host configuration to finish | 133.41s 2024-01-17 09:30:43.772286 | 000c29b1-9f06-28d0-4abe-00000000c730 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 133.31s 2024-01-17 09:30:43.773010 | 000c29b1-9f06-28d0-4abe-00000000c74d | SUMMARY | overcloud-controller-1 | Wait for puppet host configuration to finish | 133.30s 2024-01-17 09:30:43.773675 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-2 | Run puppet on the host to apply IPtables rules | 81.78s 2024-01-17 09:30:43.774343 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-0 | Run puppet on the host to apply IPtables rules | 80.99s 2024-01-17 09:30:43.774928 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-1 | Run puppet on the host to apply IPtables rules | 80.71s 2024-01-17 09:30:43.775545 | 000c29b1-9f06-28d0-4abe-00000000b7d8 | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 80.65s 2024-01-17 09:30:43.776155 | 000c29b1-9f06-28d0-4abe-00000000b8fe | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 80.21s 2024-01-17 09:30:43.776781 | 000c29b1-9f06-28d0-4abe-00000000b85e | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 76.84s 2024-01-17 09:30:43.777443 | 000c29b1-9f06-28d0-4abe-00000000b58a | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 66.95s 2024-01-17 09:30:43.778043 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-2 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 42.81s 2024-01-17 09:30:43.778660 | 000c29b1-9f06-28d0-4abe-00000000c7ab | SUMMARY | overcloud-novacompute-2 | Wait for puppet host configuration to finish | 42.09s 2024-01-17 09:30:43.779285 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-1 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 42.00s 2024-01-17 09:30:43.779870 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-0 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 40.08s 2024-01-17 09:30:43.780470 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.781183 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.781794 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.782388 | The following node(s) had failures: overcloud-controller-0, overcloud-controller-1, overcloud-controller-2 2024-01-17 09:30:43.783022 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:45.104 477248 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo5qf1b220 ] cleaned up 2024-01-17 09:30:45.104 477248 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:45.105 477248 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2024-01-17 09:30:45.793 477248 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2024-01-17 09:30:45.863 477248 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1334, in take_action 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud working_dir=self.working_dir) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1322, in take_action 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 'BlacklistedHostnames', self.working_dir)) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 405, in config_download 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=forks 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2024-01-17 09:30:47.967 477248 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.971 477248 INFO osc_lib.shell [-] END return value: 1 If I list logs on the failing container and open the /etc/config.pp file I can see that bind_port, log_file, show_image_direct_url and show_multiple_locations are successfully read but it seems they are not defined for specified class. [root@overcloud-controller-0 ~]# podman logs f319ca15fd60 include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Running puppet + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 08:25:15 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 08:25:15 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 08:25:15 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 08:25:15 puppet-user: (file & line not available) <13>Jan 17 08:25:15 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 08:25:15 puppet-user: (file & line not available) <13>Jan 17 08:25:15 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 08:25:15 puppet-user: has no parameter named 'bind_port' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'log_file' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-0.tb27.cld + rc=1 + '[' false = false ']' + set +x include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Running puppet + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 09:19:13 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 09:19:13 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 09:19:13 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 09:19:13 puppet-user: (file & line not available) <13>Jan 17 09:19:13 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 09:19:13 puppet-user: (file & line not available) <13>Jan 17 09:19:13 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 09:19:13 puppet-user: has no parameter named 'bind_port' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'log_file' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-0.tb27.cld + rc=1 + '[' false = false ']' + set +x Also the content of config.pp is as follows: include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Does anyone know what is the issue here?

2 2

[puppet][sahara] Retiring puppet-sahara
by Takashi Kajinami 18 Jan '24

18 Jan '24

Hello, As you may know, the sahara project was marked inactive during this cycle. In Puppet OpenStack project we have maintained the puppet-sahara module to deploy and manage sahara deployments. We haven't seen any feedback from actual users for a while and I doubt anyone is actually using it. # AFAIK all of the recent updates are common maintenance works. I've updated # the module to improve parameter support coverage but I've never used sahara # by myself. Based on the situation, I propose retiring this module. In case you have any concern (though in that case the concern should be directed to the sahara project itself), please let me know. Unless I hear any objections or improvements about sahara's status, I'll start the retirement process after milestone 3 (Week of Feb 26). Thank you, Takashi Kajinami irc: tkajinam

2 1

help with multiattach volume type in [cinder]
by Shubham Kumar Yadav 18 Jan '24

18 Jan '24

Hi, i have a question/issue related to multiattach feature in cinder i have an openstack cluster and i have configured multiattach in the openstack cinder now when i create 1 volume (with volume type multiattach) and attach it to 2 instances test1 and test2 on test1 sudo mkfs -t ext4 /dev/sdb sudo mkdir /mnt/mydisk sudo mount /dev/sdb /mnt/mydisk touch /mnt/mydisk/shared_file echo "This is some shared content" | sudo tee /mnt/mydisk/shared_file on test2 sudo mkdir /mnt/mydisk sudo mount /dev/sdb /mnt/mydisk ls /mnt/mydisk/ cat /mnt/mydisk/shared_file i can see the shared_file now when i add new files on test1 im not able to see it on test2 untill i unmount the volume with directory and mount the directory again to the volume on test2. what could be the solution for it? how can i make the volume share datas instantly between all the instances connected to the volume i have tried with ext4 and xfs file system Hope to hear from you soon Thank you!

2 1

Problem with deploying deployed ceph in OS wallaby.
by davorin.mikulic＠gmail.com 17 Jan '24

17 Jan '24

Hello. I have a problem deploying ceph cluster prior to deploying OS with the following command. openstack overcloud ceph deploy -o /home/stack/3XOenvironment/deployed-ceph.yaml --stack overcloud --roles-data /home/stack/3XOenvironment/30-roles_data.yaml --network-data /home/stack/3XOenvironment/40-network_data.yaml --config /home/stack/3XOenvironment/ceph.conf --ntp-server 10.100.250.101 --container-image-prepare /home/stack/3XOenvironment/00-containers-prepare-parameter.yaml /home/stack/3XOenvironment/overcloud-baremetal-deployed.yaml -vvv I have successfully provisioned overcloud networks, vips and nodes (HPE GEN 9 machines). Ceph clusters seems to create but it fails on creating the environment file (deployed-ceph.yaml) for the openstack overcloud deploy command. The error involves tld_option which I cannot find anywhere. 2024-01-16 11:29:06.616835 | 000c29b1-9f06-81af-0614-00000000011b | TASK | Ceph cluster status 2024-01-16 11:29:06.654033 | 000c29b1-9f06-81af-0614-00000000011b | OK | Ceph cluster status | overcloud-controller-0 | result={ "changed": false, "msg": [ " cluster:", " id: 472da67d-c46c-5717-880e-574b9105d723", " health: HEALTH_OK", " ", " services:", " mon: 3 daemons, quorum overcloud-controller-0,overcloud-controller-2,overcloud-controller-1 (age 25m)", " mgr: overcloud-controller-0.ewcaqz(active, since 26m), standbys: overcloud-controller-2.gaxwws, overcloud-controller-1.muacne", " osd: 3 osds: 3 up (since 23m), 3 in (since 24m)", " ", " data:", " pools: 1 pools, 1 pgs", " objects: 0 objects, 0 B", " usage: 14 MiB used, 22 TiB / 22 TiB avail", " pgs: 1 active+clean", " " ] } 2024-01-16 11:29:06.655731 | 000c29b1-9f06-81af-0614-00000000011b | TIMING | tripleo_cephadm : Ceph cluster status | overcloud-controller-0 | 0:00:45.445615 | 0.04s 2024-01-16 11:29:06.668350 | 000c29b1-9f06-81af-0614-000000000131 | TASK | Create Deployed Ceph Heat Environment file to pass to overcloud deploy <localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack <localhost> EXEC /bin/sh -c 'echo ~stack && sleep 0' <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/stack/.ansible/tmp `"&& mkdir "` echo /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570 `" && echo ansible-tmp-1705400946.7091644-595040-96765848827570="` echo /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570 `" ) && sleep 0' <localhost> EXEC /bin/sh -c 'rm -f -r /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570/ > /dev/null 2>&1 && sleep 0' 2024-01-16 11:29:06.768346 | 000c29b1-9f06-81af-0614-000000000131 | FATAL | Create Deployed Ceph Heat Environment file to pass to overcloud deploy | overcloud-controller-0 -> localhost | error={ "changed": false, "msg": "AnsibleUndefinedVariable: {{ tld_option }}: 'tld_option' is undefined" } [WARNING]: ('overcloud-controller-0 -> localhost', '000c29b1-9f06-81af-0614-000000000131') missing from stats PLAY RECAP ********************************************************************* overcloud-controller-0 : ok=37 changed=8 unreachable=0 failed=1 skipped=43 rescued=0 ignored=1 undercloud : ok=10 changed=3 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024-01-16 11:29:06.777893 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.778573 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.779338 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:45.569227 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.780170 | UUID | Info | Host | Task Name | Run Time 2024-01-16 11:29:06.781078 | 000c29b1-9f06-81af-0614-000000000187 | SUMMARY | undercloud | tripleo_run_cephadm : Run ceph-admin-user-playbook for public and private keys | 7.36s 2024-01-16 11:29:06.781746 | 000c29b1-9f06-81af-0614-000000000188 | SUMMARY | undercloud | tripleo_run_cephadm : Run ceph-admin-user-playbook for public keys | 6.19s 2024-01-16 11:29:06.782621 | 000c29b1-9f06-81af-0614-00000000009b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set public/cluster network and v4/v6 ms_bind unless already in ceph.conf | 4.41s 2024-01-16 11:29:06.783807 | 000c29b1-9f06-81af-0614-0000000000b2 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 3.20s 2024-01-16 11:29:06.784945 | 000c29b1-9f06-81af-0614-00000000002d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 3.19s 2024-01-16 11:29:06.785972 | 000c29b1-9f06-81af-0614-0000000002fb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for expected number of osds to be running | 1.27s 2024-01-16 11:29:06.786740 | 000c29b1-9f06-81af-0614-000000000118 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show the deployed daemons | 1.21s 2024-01-16 11:29:06.787585 | 000c29b1-9f06-81af-0614-00000000011a | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Print the status of the deployed Ceph cluster | 1.21s 2024-01-16 11:29:06.788369 | 000c29b1-9f06-81af-0614-000000000115 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.16s 2024-01-16 11:29:06.789142 | 000c29b1-9f06-81af-0614-00000000029f | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set container image base in ceph configuration | 1.12s 2024-01-16 11:29:06.789892 | 000c29b1-9f06-81af-0614-0000000000af | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.11s 2024-01-16 11:29:06.790586 | 000c29b1-9f06-81af-0614-00000000003b | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.89s 2024-01-16 11:29:06.791367 | 000c29b1-9f06-81af-0614-000000000184 | SUMMARY | undercloud | tripleo_run_cephadm : Populate cephadm_admin_limit_file | 0.81s 2024-01-16 11:29:06.792182 | 000c29b1-9f06-81af-0614-000000000030 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 0.69s 2024-01-16 11:29:06.792987 | 000c29b1-9f06-81af-0614-000000000019 | SUMMARY | undercloud | Create Ceph spec based on baremetal_deployed_path and tripleo_roles | 0.68s 2024-01-16 11:29:06.793962 | 000c29b1-9f06-81af-0614-00000000003e | SUMMARY | overcloud-controller-0 | push it to tripleo_cephadm_assimilate_conf on bootstrap node | 0.66s 2024-01-16 11:29:06.794719 | 000c29b1-9f06-81af-0614-000000000185 | SUMMARY | undercloud | tripleo_run_cephadm : Populate cephadm_non_admin_limit_file | 0.59s 2024-01-16 11:29:06.795418 | 000c29b1-9f06-81af-0614-00000000006e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat necessary files to bootstrap with cephadm | 0.56s 2024-01-16 11:29:06.796092 | 000c29b1-9f06-81af-0614-00000000002e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.37s 2024-01-16 11:29:06.796735 | 000c29b1-9f06-81af-0614-000000000038 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.37s 2024-01-16 11:29:06.797373 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.798022 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.798652 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.799383 | The following node(s) had failures: overcloud-controller-0 2024-01-16 11:29:06.800015 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Temporary directory [ /tmp/tripleozr0mivzo ] cleaned up Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 Temporary directory [ /tmp/tripleo9zw51ssj ] cleaned up Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v2/overcloud_ceph.py", line 739, in take_action skip_tags=','.join(skip_tags), File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook raise RuntimeError(err_msg) RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 clean_up OvercloudCephDeploy: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 END return value: 1

2 2

[heat] Propose retirement of heat-cfnclient
by Takashi Kajinami 17 Jan '24

17 Jan '24

Hello, In heat project, we have had the heat-cfnclient library which contains the client utilities to consume AWS-compatible APIs in heat. However this repository is not properly maintained and I doubt that anyone is using (or even can use) it with recent Heat. - The repository still contains the utility for CloudWatch API which was already removed from heat - The heat-cfn command relies on Keystone v2 instead of v3 - No release has been made since the repository was created Based on the current situation I'd propose retiring the repository. I'd appreciate any feedback and if I hear no objections then I'll start the official process to retire the repository. Thank you, Takashi

2 3

[kolla-ansible][Antelope][Senlin] Senlin Rolling Updates Not Working
by Devin Patterson 16 Jan '24

16 Jan '24

Hello, We have deployed Senlin clusters to manage VM health. When attempting to do rolling updates of VMs by updating the profile, the VMs go into an error state with the following error in the senlin engine logs. I've included the Heat template and OS commands. Any help would be appreciated. openstack cluster update --profile <new_profile> <senlin_cluster> cluster event show <event_d> status_reason | Failed in updating nodes. heat_template_version: 2021-04-16 description: OpenStack Heat template to deploy a Senlin autoscaling group resources: profile1: type: OS::Senlin::Profile properties: type: os.nova.server-1.0 properties: flavor: m1.small image: cirros-0.5.1 key_name: keypair networks: - network: network user_data: | #!/bin/sh echo 'hello, world' > /tmp/test_file batch_policy: type: OS::Senlin::Policy properties: type: senlin.policy.batch-1.0 bindings: - cluster: {get_resource: cluster} properties: min_in_service: 1 max_batch_size: 2 pause_time: 3 cluster: type: OS::Senlin::Cluster properties: desired_capacity: 8 min_size: 2 profile: { get_resource: profile1 } receiver_scale_out: type: OS::Senlin::Receiver properties: cluster: {get_resource: cluster} action: CLUSTER_SCALE_OUT type: webhook receiver_scale_in: type: OS::Senlin::Receiver properties: cluster: {get_resource: cluster} action: CLUSTER_SCALE_IN type: webhook outputs: receiver_scale_out: description: Webhook to scale out cluster. value: str_replace: template: curl -X POST LINK params: LINK: {get_attr: [receiver_scale_out, channel, alarm_url]} receiver_scale_in: description: Webhook to scale in cluster. value: str_replace: template: curl -X POST LINK params: LINK: {get_attr: [receiver_scale_in, channel, alarm_url]} Regards, Devin

1 0

[kolla-ansible][Antelope][Heat] Heat Autoscaling Scaling Policy URL
by Devin Patterson 16 Jan '24

16 Jan '24

Hello, I am attempting to create an auto-scaling group with Heat. The group is deployed properly along with scale up and scale down policies. When attempting to curl with POST, I get the following response. This is using the "alarm_url" from the scaling policy. Below is also the Heat template being utilized. Below the error response is the curl command. Wondering why there is an AWSAccessKeyId in the signature of the URL as well. <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> CURL COMMAND: curl -X POST <external-os-url>/v1/signal/arn%3Aopenstack%3Aheat%3A%omitted3Astacks/stack_1/omitted/resources/scale_down_policy\?SignatureMethod\=HmacSHA256\&SignatureVersion\=2\&AWSAccessKeyId\=omitted\&Signature\=omitted TEMPLATE: heat_template_version: 2021-04-16 description: OpenStack Heat template to deploy a autoscaling group resources: asg: type: OS::Heat::AutoScalingGroup properties: desired_capacity: 6 max_size: 8 min_size: 2 rolling_updates: min_in_service: 1 max_batch_size: 2 pause_time: 3 resource: type: OS::Nova::Server properties: flavor: m1.small image: image key_name: my-keypair networks: - network: network user_data: | #!/bin/sh echo 'hello, world123' > /tmp/test_file scale_up_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: { get_resource: asg } cooldown: 60 scaling_adjustment: 1 scale_down_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: { get_resource: asg } cooldown: 60 scaling_adjustment: -1 outputs: scale_up_url: description: > This URL is the webhook to scale up the group. You can invoke the scale-up operation by doing an HTTP POST to this URL; no body nor extra headers are needed. value: {get_attr: [scale_up_policy, alarm_url]} scale_dn_url: description: > This URL is the webhook to scale down the group. You can invoke the scale-down operation by doing an HTTP POST to this URL; no body nor extra headers are needed. value: {get_attr: [scale_down_policy, alarm_url]} asg_size: description: > This is the current size of the auto scaling group. value: {get_attr: [asg, current_size]} server_list: description: > This is a list of server names that are part of the group. value: {get_attr: [asg, outputs_list, name]} networks: description: > This is a map of server resources and their networks. value: {get_attr: [asg, outputs, networks]} server_ips: description: > This is a list of first ip addresses of the servers in the group for a specified network. value: {get_attr: [asg, outputs_list, networks, {get_param: network}, 0]} Regards, Devin

1 0

[magnum] k8s-keystone-auth pod CrashLoopBackOff
by Eugen Block 16 Jan '24

16 Jan '24

Hi there, I've noticed several magnum related threads and thought I'd give it a try while I keep digging. I've only recently started to work with magnum, and this is how far I get. Note that this openstack is still running on version Victoria, I'm working towards migrating it to a newer release. My coe cluster template currently looks like this (I'll omit the defaults): | labels | {'kube_tag': 'v1.18.2', 'hyperkube_prefix': 'k8s.gcr.io/'} | | https_proxy | http://<IP>:<PORT> | | http_proxy | http://<IP>:<PORT> | | image_id | fedora-coreos33 | | dns_nameserver | <MY_DNS> | I've already tried quite a lot of different kube_tags and hyperkube_prefixes, this combination gets me at least to have some running containers: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7fbe3f8f308f docker.io/openstackmagnum/heat-container-agent:victoria-stable-1 /usr/bin/start-he... 42 minutes ago Up 42 minutes ago heat-container-agent a8b6ae357034 k8s.gcr.io/hyperkube:v1.18.2 kube-apiserver --... 40 minutes ago Up 40 minutes ago kube-apiserver cb7774fbf849 k8s.gcr.io/hyperkube:v1.18.2 kube-controller-m... 40 minutes ago Up 40 minutes ago kube-controller-manager 448f1c02cbd4 k8s.gcr.io/hyperkube:v1.18.2 kube-scheduler --... 40 minutes ago Up 40 minutes ago kube-scheduler 4be28a0a2271 k8s.gcr.io/hyperkube:v1.18.2 kubelet --logtost... 40 minutes ago Up 40 minutes ago kubelet 542ffdb081e7 k8s.gcr.io/hyperkube:v1.18.2 kube-proxy --logt... 40 minutes ago Up 40 minutes ago kube-proxy 61874a4e4595 quay.io/coreos/etcd:v3.4.6 /usr/local/bin/et... 40 minutes ago Up 40 minutes ago etcd But not all pods are starting: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl get pods -o wide --kubeconfig=/etc/kubernetes/admin.conf -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-786ffb7797-dpkbx 1/1 Running 0 25m 10.100.0.3 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system coredns-786ffb7797-mslfk 1/1 Running 0 25m 10.100.0.2 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system dashboard-metrics-scraper-6b4884c9d5-nrvbg 1/1 Running 0 25m 10.100.0.5 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system k8s-keystone-auth-qt7n5 0/1 CrashLoopBackOff 4 5m12s 10.0.0.116 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system kube-dns-autoscaler-75859754fd-69zxx 0/1 Pending 0 25m <none> <none> <none> <none> kube-system kube-flannel-ds-cn5jf 1/1 Running 0 25m 10.0.0.116 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system kubernetes-dashboard-c98496485-qw4hl 1/1 Running 0 25m 10.100.0.4 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system magnum-metrics-server-79556d6999-hkqcw 0/1 Pending 0 25m <none> <none> <none> <none> I didn't look too deep into the two other failing pods, I think the keystone pod is more important at this point or maybe even is required for the other pods to start, I don't know yet. I can pull the image manually and the "describe pod" command also shows that the container was pulled successfully, but I can't find other hints what is actually failing. Here is the full output of "describe pod": ---snip--- [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system describe pod/k8s-keystone-auth-qt7n5 Name: k8s-keystone-auth-qt7n5 Namespace: kube-system Priority: 0 Node: kubernetes-cluster20-buuxqog6cfnn-master-0/10.0.0.116 Start Time: Mon, 15 Jan 2024 09:13:16 +0000 Labels: controller-revision-hash=6875f96b46 k8s-app=k8s-keystone-auth pod-template-generation=1 Annotations: kubernetes.io/psp: magnum.privileged Status: Running IP: 10.0.0.116 IPs: IP: 10.0.0.116 Controlled By: DaemonSet/k8s-keystone-auth Containers: k8s-keystone-auth: Container ID: docker://adca0b9a9eb011f22530886a42bf862b4cfa7e6b7965cd77d6dac4f37222b827 Image: docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0 Image ID: docker-pullable://k8scloudprovider/k8s-keystone-auth@sha256:ab72a7e5b9eca9af2762796690be800bce2de25dee8bd5ad55d0996b29c85146 Port: 8443/TCP Host Port: 8443/TCP Args: ./bin/k8s-keystone-auth --tls-cert-file /etc/kubernetes/certs/server.crt --tls-private-key-file /etc/kubernetes/certs/server.key --policy-configmap-name k8s-keystone-auth-policy --keystone-url http://<CONTROLLER>:5000/v3 --sync-configmap-name keystone-sync-policy --keystone-ca-file /etc/kubernetes/ca-bundle.crt --listen 127.0.0.1:8443 State: Terminated Reason: Error Exit Code: 1 Started: Mon, 15 Jan 2024 09:23:52 +0000 Finished: Mon, 15 Jan 2024 09:23:52 +0000 Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 15 Jan 2024 09:18:41 +0000 Finished: Mon, 15 Jan 2024 09:18:41 +0000 Ready: False Restart Count: 6 Requests: cpu: 200m Environment: <none> Mounts: /etc/kubernetes from ca-certs (ro) /etc/kubernetes/certs from k8s-certs (ro) /var/run/secrets/kubernetes.io/serviceaccount from k8s-keystone-auth-token-vhm4r (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: k8s-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/certs HostPathType: DirectoryOrCreate ca-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes HostPathType: DirectoryOrCreate k8s-keystone-auth-token-vhm4r: Type: Secret (a volume populated by a Secret) SecretName: k8s-keystone-auth-token-vhm4r Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: :NoSchedule :NoExecute CriticalAddonsOnly node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/network-unavailable:NoSchedule node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/pid-pressure:NoSchedule node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unschedulable:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/k8s-keystone-auth-qt7n5 to kubernetes-cluster20-buuxqog6cfnn-master-0 Warning Failed 9m52s kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Error: ErrImagePull Warning Failed 9m52s kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Failed to pull image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0": rpc error: code = Unknown desc = Get "https://registry-1.docker.io/v2/k8scloudprovider/k8s-keystone-auth/manifest…": Internal Server Error Normal Pulling 9m37s (x5 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Pulling image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0" Normal Pulled 9m36s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Successfully pulled image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0" Normal Created 9m35s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Created container k8s-keystone-auth Normal Started 9m35s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Started container k8s-keystone-auth Warning BackOff 33s (x48 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Back-off restarting failed container ---snip--- I'm obviously not allowed to get logs from the pod: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system logs pod/k8s-keystone-auth-qt7n5 Error from server: Get https://10.0.0.116:10250/containerLogs/kube-system/k8s-keystone-auth-qt7n5/…: Forbidden I appreciate any hints! Thanks! Eugen

1 3

[keystone] Keystone as a Service Provider, 401 - Unauthorized Exception for SSO
by joeldag7＠gmail.com 16 Jan '24

16 Jan '24

We're currently working on implementing SSO for our university's OpenStack environment using OpenStack Ansible (Yoga version). Our goal is to allow students to log in with their university credentials via an Identity Provider - currently we're using our own Keycloak instance as IdP. The plan is to automatically create a new project for first time users, while returning students will be redirected to their existing projects to manage their instances. For initial testing, we're directing all users to a single project. Eventually, we would like to script the dynamic creation of projects on the controller node for first-time logins later. However, we've run into some problems: Our Keystone configuration in the user_variables.yml file is as follows: ``` keystone_sp: apache_mod: 'mod_auth_openidc' cadf_notifications: true trusted_dashboard_list: - "https://openstack-example.com/auth/websso/" - "http://example.keycloak.com/realms/openstack*" - "https://openstack-example.com/auth/login/" trusted_idp_list: - name: "oidc-idp" oidc_provider_metadata_url: http://example.keycloak.com/realms/openstack/.well-known/openid-configurati… oidc_client_id: ******** oidc_client_secret: ********* oidc_crypto_passphrase: ****** oidc_redirect_uri: https://openstack-example.com/v3/OS-FEDERATION/identity_providers/oidc-idp/… entity_ids: - 'http://exmaple.keycloak.com/realms/openstack' federated_identities: - domain: default project: fedproject group: fedgroup role: member protocols: - name: openid mapping: name: openid-mapping rules: - remote: - type: email local: - group: name: fedgroup domain: name: Default user: name: '{0}' ``` This setup is based on the openstack documentation (https://docs.openstack.org/openstack-ansible-os_keystone/yoga/configure-fed…) After many attempts with different configurations, this one successfully redirects to our Keycloak IdP for authentication. However, after login, when redirected back to OpenStack, we get a 401 Unauthorised error stating "The request you have made requires authentication". The SAML tracer logs confirm that Keycloak issues a valid token along with the user information. Despite this, we find no additional error details in the Keystone container, Apache2 and controller logs. This leads me to believe that the problem may lie in the mapping configuration, although it appears to work when tested with the keystone-manage mapping-engine tool. But still, the doc (https://docs.openstack.org/keystone/queens/advanced-topics/federation/mappi…) claims that a 401 exception could be caused by the mapping. Any insight or suggestions on how to solve this problem would be greatly appreciated, particularly regarding the mapping process during authentication. I'm also looking for advice on how to extract more detailed error information. At the moment, the logs don't reveal much beyond the 401 error message. Am I missing something or is this all the information OpenStack typically provides in such cases? Thank you for your help and guidance.

1 0