- openstack-discuss - lists.openstack.org

[neutron] Neutron drivers meeting
by Brian Haley 18 Jan '24

18 Jan '24

Hello Neutrinos, Please remember tomorrow (Friday) we have a Neutron drivers meeting. This is the agenda: https://wiki.openstack.org/wiki/Meetings/NeutronDrivers There are currently two topics: * (lajoskatona): quicky discuss the eventlet hell and what are the first steps: https://review.opendev.org/q/topic:bump-eventlet * (mtomaska): https://bugs.launchpad.net/neutron/+bug/2047849. [RFE] Start using oslo messaging namespaces https://review.opendev.org/c/openstack/neutron/+/905309 I realize there might be a couple of people out, if we do not have quorum will just postpone until next week. Thanks, -Brian

1 0

Problem with failing container-puppet-glance_api_internal during wallaby overcloud deploy.
by davorin.mikulic＠gmail.com 18 Jan '24

18 Jan '24

Hi, I have successfully deployed networks, vips, overcloud nodes and ceph cluster. Now I am proceeding with openstack overcloud deploy command as follows: openstack overcloud deploy --deployed-server --answers-file /home/stack/3XOenvironment/answers.yaml answers.yaml is as follows: (undercloud) [stack@uc4tb27 3XOenvironment]$ cat answers.yaml templates: /home/stack/generated-openstack-tripleo-heat-templates networks: /home/stack/3XOenvironment/40-network_data.yaml roles: /home/stack/3XOenvironment/30-roles_data.yaml environments: - /home/stack/3XOenvironment/deployed-server-environment.yaml - /home/stack/3XOenvironment/network-environment.yaml - /home/stack/3XOenvironment/overcloud-networks-deployed.yaml - /home/stack/3XOenvironment/overcloud-baremetal-deployed.yaml - /home/stack/3XOenvironment/deployed-vips.yaml - /home/stack/3XOenvironment/00-containers-prepare-parameter.yaml - /home/stack/3XOenvironment/10-docker-ha.yaml - /home/stack/3XOenvironment/cephadm.yaml - /home/stack/3XOenvironment/deployed-ceph.yaml We are not using a TLS setup anywhere in the overcloud or the undercloud. During the deploy, after the stack has been successfully created and deploy proceeds to puppet configuration of containers in step 1, I am getting the following error: 2024-01-17 09:30:39.360378 | 000c29b1-9f06-28d0-4abe-00000000d389 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-1 | 0:18:04.150199 | 324.03s 2024-01-17 09:30:43.693815 | | WARNING | ERROR: Can't run container container-puppet-glance_api_internal stderr: + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 08:25:20 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 08:25:20 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 08:25:20 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 08:25:20 puppet-user: (file & line not available) <13>Jan 17 08:25:20 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 08:25:20 puppet-user: (file & line not available) <13>Jan 17 08:25:20 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 08:25:20 puppet-user: has no parameter named 'bind_port' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'log_file' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 08:25:20 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-2.tb27.cld + rc=1 + '[' false = false ']' + set +x 2024-01-17 09:30:43.695467 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | FATAL | Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-2 | error={"changed": false, "msg": "Failed containers: container-puppet-glance_api_internal"} 2024-01-17 09:30:43.696471 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | TIMING | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | overcloud-controller-2 | 0:18:08.486292 | 323.73s PLAY RECAP ********************************************************************* localhost : ok=1 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0 overcloud-cephstorage-0 : ok=238 changed=96 unreachable=0 failed=0 skipped=135 rescued=0 ignored=1 overcloud-cephstorage-1 : ok=240 changed=96 unreachable=0 failed=0 skipped=130 rescued=0 ignored=1 overcloud-cephstorage-2 : ok=240 changed=96 unreachable=0 failed=0 skipped=130 rescued=0 ignored=1 overcloud-controller-0 : ok=295 changed=140 unreachable=0 failed=1 skipped=175 rescued=0 ignored=1 overcloud-controller-1 : ok=306 changed=140 unreachable=0 failed=1 skipped=164 rescued=0 ignored=1 overcloud-controller-2 : ok=306 changed=140 unreachable=0 failed=1 skipped=164 rescued=0 ignored=1 overcloud-novacompute-0 : ok=281 changed=132 unreachable=0 failed=0 skipped=165 rescued=0 ignored=1 overcloud-novacompute-1 : ok=288 changed=132 unreachable=0 failed=0 skipped=158 rescued=0 ignored=1 overcloud-novacompute-2 : ok=288 changed=132 unreachable=0 failed=0 skipped=158 rescued=0 ignored=1 undercloud : ok=25 changed=9 unreachable=0 failed=0 skipped=2 rescued=0 ignored=1 2024-01-17 09:30:43.764701 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.765549 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 1578 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.766211 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:18:08.556038 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.766837 | UUID | Info | Host | Task Name | Run Time 2024-01-17 09:30:43.767508 | 000c29b1-9f06-28d0-4abe-00000000d389 | SUMMARY | overcloud-controller-1 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 324.03s 2024-01-17 09:30:43.768148 | 000c29b1-9f06-28d0-4abe-00000000d33c | SUMMARY | overcloud-controller-0 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 323.73s 2024-01-17 09:30:43.768828 | 000c29b1-9f06-28d0-4abe-00000000d4d9 | SUMMARY | overcloud-controller-2 | tripleo_container_manage : Create containers managed by Podman for /var/lib/tripleo-config/container-puppet-config/step_1 | 323.73s 2024-01-17 09:30:43.769558 | 000c29b1-9f06-28d0-4abe-00000000b974 | SUMMARY | overcloud-novacompute-0 | Pre-fetch all the containers | 181.76s 2024-01-17 09:30:43.770215 | 000c29b1-9f06-28d0-4abe-00000000b9f7 | SUMMARY | overcloud-novacompute-1 | Pre-fetch all the containers | 175.41s 2024-01-17 09:30:43.770929 | 000c29b1-9f06-28d0-4abe-00000000ba70 | SUMMARY | overcloud-novacompute-2 | Pre-fetch all the containers | 169.84s 2024-01-17 09:30:43.771586 | 000c29b1-9f06-28d0-4abe-00000000c777 | SUMMARY | overcloud-controller-2 | Wait for puppet host configuration to finish | 133.41s 2024-01-17 09:30:43.772286 | 000c29b1-9f06-28d0-4abe-00000000c730 | SUMMARY | overcloud-controller-0 | Wait for puppet host configuration to finish | 133.31s 2024-01-17 09:30:43.773010 | 000c29b1-9f06-28d0-4abe-00000000c74d | SUMMARY | overcloud-controller-1 | Wait for puppet host configuration to finish | 133.30s 2024-01-17 09:30:43.773675 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-2 | Run puppet on the host to apply IPtables rules | 81.78s 2024-01-17 09:30:43.774343 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-0 | Run puppet on the host to apply IPtables rules | 80.99s 2024-01-17 09:30:43.774928 | 000c29b1-9f06-28d0-4abe-000000003b66 | SUMMARY | overcloud-controller-1 | Run puppet on the host to apply IPtables rules | 80.71s 2024-01-17 09:30:43.775545 | 000c29b1-9f06-28d0-4abe-00000000b7d8 | SUMMARY | overcloud-controller-0 | Pre-fetch all the containers | 80.65s 2024-01-17 09:30:43.776155 | 000c29b1-9f06-28d0-4abe-00000000b8fe | SUMMARY | overcloud-controller-2 | Pre-fetch all the containers | 80.21s 2024-01-17 09:30:43.776781 | 000c29b1-9f06-28d0-4abe-00000000b85e | SUMMARY | overcloud-controller-1 | Pre-fetch all the containers | 76.84s 2024-01-17 09:30:43.777443 | 000c29b1-9f06-28d0-4abe-00000000b58a | SUMMARY | undercloud | tripleo_container_image_prepare : Container image prepare | 66.95s 2024-01-17 09:30:43.778043 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-2 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 42.81s 2024-01-17 09:30:43.778660 | 000c29b1-9f06-28d0-4abe-00000000c7ab | SUMMARY | overcloud-novacompute-2 | Wait for puppet host configuration to finish | 42.09s 2024-01-17 09:30:43.779285 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-1 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 42.00s 2024-01-17 09:30:43.779870 | 000c29b1-9f06-28d0-4abe-00000000b78c | SUMMARY | overcloud-controller-0 | tripleo_container_tag : Pull uc4tb27.ctlplane.tb27.cld:8787/tripleowallaby/openstack-cinder-volume:current-tripleo image | 40.08s 2024-01-17 09:30:43.780470 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.781183 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.781794 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 3 ~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:43.782388 | The following node(s) had failures: overcloud-controller-0, overcloud-controller-1, overcloud-controller-2 2024-01-17 09:30:43.783022 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-17 09:30:45.104 477248 INFO tripleoclient.utils.utils [-] Temporary directory [ /tmp/tripleo5qf1b220 ] cleaned up 2024-01-17 09:30:45.104 477248 ERROR tripleoclient.utils.utils [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:45.105 477248 WARNING tripleoclient.utils.safe_write [-] The output file /home/stack/overcloud-deploy/overcloud/overcloud-deployment_status.yaml will be overriden: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh Host 10.100.27.127 not found in /home/stack/.ssh/known_hosts Overcloud Endpoint: http://10.100.27.127:5000 Overcloud Horizon Dashboard URL: http://10.100.27.127:80/dashboard Overcloud rc file: /home/stack/overcloud-deploy/overcloud/overcloudrc and /home/stack/overcloudrc Overcloud Deployed with error 2024-01-17 09:30:45.793 477248 INFO tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Stopping ephemeral heat. 2024-01-17 09:30:45.863 477248 INFO tripleoclient.heat_launcher [-] Pod does not exist: ephemeral-heat 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud [-] Exception occured while running the command: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud Traceback (most recent call last): 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud super(Command, self).run(parsed_args) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return super(Command, self).run(parsed_args) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud return_code = self.take_action(parsed_args) or 0 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1334, in take_action 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud working_dir=self.working_dir) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 227, in __exit__ 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud self.force_reraise() 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 200, in force_reraise 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise self.value 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_deploy.py", line 1322, in take_action 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 'BlacklistedHostnames', self.working_dir)) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/deployment.py", line 405, in config_download 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud forks=forks 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud raise RuntimeError(err_msg) 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.952 477248 ERROR tripleoclient.v1.overcloud_deploy.DeployOvercloud 2024-01-17 09:30:47.967 477248 ERROR openstack [-] Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh: RuntimeError: Ansible execution failed. playbook: /home/stack/overcloud-deploy/overcloud/config-download/overcloud/deploy_steps_playbook.yaml, Run Status: failed, Return Code: 2, To rerun the failed command manually execute the following script: /home/stack/overcloud-deploy/overcloud/config-download/ansible-playbook-command.sh 2024-01-17 09:30:47.971 477248 INFO osc_lib.shell [-] END return value: 1 If I list logs on the failing container and open the /etc/config.pp file I can see that bind_port, log_file, show_image_direct_url and show_multiple_locations are successfully read but it seems they are not defined for specified class. [root@overcloud-controller-0 ~]# podman logs f319ca15fd60 include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Running puppet + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 08:25:15 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 08:25:15 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 08:25:15 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 08:25:15 puppet-user: (file & line not available) <13>Jan 17 08:25:15 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 08:25:15 puppet-user: (file & line not available) <13>Jan 17 08:25:15 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 08:25:15 puppet-user: has no parameter named 'bind_port' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'log_file' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 08:25:15 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-0.tb27.cld + rc=1 + '[' false = false ']' + set +x include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Running puppet + /usr/bin/puppet apply --summarize --detailed-exitcodes --color=false --modulepath=/etc/puppet/modules:/usr/share/openstack-puppet/modules --tags '"file,file_line,concat,augeas,cron,glance_api_config,glance_api_paste_ini,glance_swift_config,glance_cache_config,glance_image_import_config"' /etc/config.pp + logger -s -t puppet-user <13>Jan 17 09:19:13 puppet-user: Warning: /etc/puppet/hiera.yaml: Use of 'hiera.yaml' version 3 is deprecated. It should be converted to version 5 <13>Jan 17 09:19:13 puppet-user: (file: /etc/puppet/hiera.yaml) <13>Jan 17 09:19:13 puppet-user: Warning: Undefined variable '::deploy_config_name'; <13>Jan 17 09:19:13 puppet-user: (file & line not available) <13>Jan 17 09:19:13 puppet-user: Warning: The function 'hiera' is deprecated in favor of using 'lookup'. See https://puppet.com/docs/puppet/7.6/deprecated_language.html <13>Jan 17 09:19:13 puppet-user: (file & line not available) <13>Jan 17 09:19:13 puppet-user: Error: Evaluation Error: Error while evaluating a Resource Statement, Class[Tripleo::Profile::Base::Glance::Api]: <13>Jan 17 09:19:13 puppet-user: has no parameter named 'bind_port' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'log_file' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'show_image_direct_url' <13>Jan 17 09:19:13 puppet-user: has no parameter named 'show_multiple_locations' (file: /etc/config.pp, line: 2, column: 1) on node overcloud-controller-0.tb27.cld + rc=1 + '[' false = false ']' + set +x Also the content of config.pp is as follows: include ::tripleo::packages class { 'tripleo::profile::base::glance::api': bind_port => 9293, tls_proxy_port => 9293, log_file => '/var/log/glance/api_internal.log', show_image_direct_url => true, show_multiple_locations => true, } include tripleo::profile::base::database::mysql::client Does anyone know what is the issue here?

2 2

[puppet][sahara] Retiring puppet-sahara
by Takashi Kajinami 18 Jan '24

18 Jan '24

Hello, As you may know, the sahara project was marked inactive during this cycle. In Puppet OpenStack project we have maintained the puppet-sahara module to deploy and manage sahara deployments. We haven't seen any feedback from actual users for a while and I doubt anyone is actually using it. # AFAIK all of the recent updates are common maintenance works. I've updated # the module to improve parameter support coverage but I've never used sahara # by myself. Based on the situation, I propose retiring this module. In case you have any concern (though in that case the concern should be directed to the sahara project itself), please let me know. Unless I hear any objections or improvements about sahara's status, I'll start the retirement process after milestone 3 (Week of Feb 26). Thank you, Takashi Kajinami irc: tkajinam

2 1

help with multiattach volume type in [cinder]
by Shubham Kumar Yadav 18 Jan '24

18 Jan '24

Hi, i have a question/issue related to multiattach feature in cinder i have an openstack cluster and i have configured multiattach in the openstack cinder now when i create 1 volume (with volume type multiattach) and attach it to 2 instances test1 and test2 on test1 sudo mkfs -t ext4 /dev/sdb sudo mkdir /mnt/mydisk sudo mount /dev/sdb /mnt/mydisk touch /mnt/mydisk/shared_file echo "This is some shared content" | sudo tee /mnt/mydisk/shared_file on test2 sudo mkdir /mnt/mydisk sudo mount /dev/sdb /mnt/mydisk ls /mnt/mydisk/ cat /mnt/mydisk/shared_file i can see the shared_file now when i add new files on test1 im not able to see it on test2 untill i unmount the volume with directory and mount the directory again to the volume on test2. what could be the solution for it? how can i make the volume share datas instantly between all the instances connected to the volume i have tried with ext4 and xfs file system Hope to hear from you soon Thank you!

2 1

Problem with deploying deployed ceph in OS wallaby.
by davorin.mikulic＠gmail.com 17 Jan '24

17 Jan '24

Hello. I have a problem deploying ceph cluster prior to deploying OS with the following command. openstack overcloud ceph deploy -o /home/stack/3XOenvironment/deployed-ceph.yaml --stack overcloud --roles-data /home/stack/3XOenvironment/30-roles_data.yaml --network-data /home/stack/3XOenvironment/40-network_data.yaml --config /home/stack/3XOenvironment/ceph.conf --ntp-server 10.100.250.101 --container-image-prepare /home/stack/3XOenvironment/00-containers-prepare-parameter.yaml /home/stack/3XOenvironment/overcloud-baremetal-deployed.yaml -vvv I have successfully provisioned overcloud networks, vips and nodes (HPE GEN 9 machines). Ceph clusters seems to create but it fails on creating the environment file (deployed-ceph.yaml) for the openstack overcloud deploy command. The error involves tld_option which I cannot find anywhere. 2024-01-16 11:29:06.616835 | 000c29b1-9f06-81af-0614-00000000011b | TASK | Ceph cluster status 2024-01-16 11:29:06.654033 | 000c29b1-9f06-81af-0614-00000000011b | OK | Ceph cluster status | overcloud-controller-0 | result={ "changed": false, "msg": [ " cluster:", " id: 472da67d-c46c-5717-880e-574b9105d723", " health: HEALTH_OK", " ", " services:", " mon: 3 daemons, quorum overcloud-controller-0,overcloud-controller-2,overcloud-controller-1 (age 25m)", " mgr: overcloud-controller-0.ewcaqz(active, since 26m), standbys: overcloud-controller-2.gaxwws, overcloud-controller-1.muacne", " osd: 3 osds: 3 up (since 23m), 3 in (since 24m)", " ", " data:", " pools: 1 pools, 1 pgs", " objects: 0 objects, 0 B", " usage: 14 MiB used, 22 TiB / 22 TiB avail", " pgs: 1 active+clean", " " ] } 2024-01-16 11:29:06.655731 | 000c29b1-9f06-81af-0614-00000000011b | TIMING | tripleo_cephadm : Ceph cluster status | overcloud-controller-0 | 0:00:45.445615 | 0.04s 2024-01-16 11:29:06.668350 | 000c29b1-9f06-81af-0614-000000000131 | TASK | Create Deployed Ceph Heat Environment file to pass to overcloud deploy <localhost> ESTABLISH LOCAL CONNECTION FOR USER: stack <localhost> EXEC /bin/sh -c 'echo ~stack && sleep 0' <localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /home/stack/.ansible/tmp `"&& mkdir "` echo /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570 `" && echo ansible-tmp-1705400946.7091644-595040-96765848827570="` echo /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570 `" ) && sleep 0' <localhost> EXEC /bin/sh -c 'rm -f -r /home/stack/.ansible/tmp/ansible-tmp-1705400946.7091644-595040-96765848827570/ > /dev/null 2>&1 && sleep 0' 2024-01-16 11:29:06.768346 | 000c29b1-9f06-81af-0614-000000000131 | FATAL | Create Deployed Ceph Heat Environment file to pass to overcloud deploy | overcloud-controller-0 -> localhost | error={ "changed": false, "msg": "AnsibleUndefinedVariable: {{ tld_option }}: 'tld_option' is undefined" } [WARNING]: ('overcloud-controller-0 -> localhost', '000c29b1-9f06-81af-0614-000000000131') missing from stats PLAY RECAP ********************************************************************* overcloud-controller-0 : ok=37 changed=8 unreachable=0 failed=1 skipped=43 rescued=0 ignored=1 undercloud : ok=10 changed=3 unreachable=0 failed=0 skipped=10 rescued=0 ignored=0 2024-01-16 11:29:06.777893 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.778573 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Total Tasks: 102 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.779338 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Elapsed Time: 0:00:45.569227 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.780170 | UUID | Info | Host | Task Name | Run Time 2024-01-16 11:29:06.781078 | 000c29b1-9f06-81af-0614-000000000187 | SUMMARY | undercloud | tripleo_run_cephadm : Run ceph-admin-user-playbook for public and private keys | 7.36s 2024-01-16 11:29:06.781746 | 000c29b1-9f06-81af-0614-000000000188 | SUMMARY | undercloud | tripleo_run_cephadm : Run ceph-admin-user-playbook for public keys | 6.19s 2024-01-16 11:29:06.782621 | 000c29b1-9f06-81af-0614-00000000009b | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Set public/cluster network and v4/v6 ms_bind unless already in ceph.conf | 4.41s 2024-01-16 11:29:06.783807 | 000c29b1-9f06-81af-0614-0000000000b2 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Apply spec | 3.20s 2024-01-16 11:29:06.784945 | 000c29b1-9f06-81af-0614-00000000002d | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Install cephadm package | 3.19s 2024-01-16 11:29:06.785972 | 000c29b1-9f06-81af-0614-0000000002fb | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Wait for expected number of osds to be running | 1.27s 2024-01-16 11:29:06.786740 | 000c29b1-9f06-81af-0614-000000000118 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Show the deployed daemons | 1.21s 2024-01-16 11:29:06.787585 | 000c29b1-9f06-81af-0614-00000000011a | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Print the status of the deployed Ceph cluster | 1.21s 2024-01-16 11:29:06.788369 | 000c29b1-9f06-81af-0614-000000000115 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.16s 2024-01-16 11:29:06.789142 | 000c29b1-9f06-81af-0614-00000000029f | SUMMARY | overcloud-controller-0 | tripleo_cephadm : set container image base in ceph configuration | 1.12s 2024-01-16 11:29:06.789892 | 000c29b1-9f06-81af-0614-0000000000af | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Get the ceph orchestrator status | 1.11s 2024-01-16 11:29:06.790586 | 000c29b1-9f06-81af-0614-00000000003b | SUMMARY | overcloud-controller-0 | push tripleo_cephadm_spec to bootstrap node if spec file exsits | 0.89s 2024-01-16 11:29:06.791367 | 000c29b1-9f06-81af-0614-000000000184 | SUMMARY | undercloud | tripleo_run_cephadm : Populate cephadm_admin_limit_file | 0.81s 2024-01-16 11:29:06.792182 | 000c29b1-9f06-81af-0614-000000000030 | SUMMARY | overcloud-controller-0 | tripleo_cephadm : List Ceph daemon instances on this host | 0.69s 2024-01-16 11:29:06.792987 | 000c29b1-9f06-81af-0614-000000000019 | SUMMARY | undercloud | Create Ceph spec based on baremetal_deployed_path and tripleo_roles | 0.68s 2024-01-16 11:29:06.793962 | 000c29b1-9f06-81af-0614-00000000003e | SUMMARY | overcloud-controller-0 | push it to tripleo_cephadm_assimilate_conf on bootstrap node | 0.66s 2024-01-16 11:29:06.794719 | 000c29b1-9f06-81af-0614-000000000185 | SUMMARY | undercloud | tripleo_run_cephadm : Populate cephadm_non_admin_limit_file | 0.59s 2024-01-16 11:29:06.795418 | 000c29b1-9f06-81af-0614-00000000006e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat necessary files to bootstrap with cephadm | 0.56s 2024-01-16 11:29:06.796092 | 000c29b1-9f06-81af-0614-00000000002e | SUMMARY | overcloud-controller-0 | tripleo_cephadm : Stat cephadm file | 0.37s 2024-01-16 11:29:06.796735 | 000c29b1-9f06-81af-0614-000000000038 | SUMMARY | overcloud-controller-0 | Ensure tripleo_cephadm_config_home (e.g. /etc/ceph) exists | 0.37s 2024-01-16 11:29:06.797373 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ End Summary Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.798022 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ State Information ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.798652 | ~~~~~~~~~~~~~~~~~~ Number of nodes which did not deploy successfully: 1 ~~~~~~~~~~~~~~~~~ 2024-01-16 11:29:06.799383 | The following node(s) had failures: overcloud-controller-0 2024-01-16 11:29:06.800015 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Temporary directory [ /tmp/tripleozr0mivzo ] cleaned up Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 Temporary directory [ /tmp/tripleo9zw51ssj ] cleaned up Exception occured while running the command Traceback (most recent call last): File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 39, in run return super(Command, self).run(parsed_args) File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run return_code = self.take_action(parsed_args) or 0 File "/usr/lib/python3.6/site-packages/tripleoclient/v2/overcloud_ceph.py", line 739, in take_action skip_tags=','.join(skip_tags), File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 775, in run_ansible_playbook raise RuntimeError(err_msg) RuntimeError: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 clean_up OvercloudCephDeploy: Ansible execution failed. playbook: /usr/share/ansible/tripleo-playbooks/cli-deployed-ceph.yaml, Run Status: failed, Return Code: 2 END return value: 1

2 2

[heat] Propose retirement of heat-cfnclient
by Takashi Kajinami 16 Jan '24

16 Jan '24

Hello, In heat project, we have had the heat-cfnclient library which contains the client utilities to consume AWS-compatible APIs in heat. However this repository is not properly maintained and I doubt that anyone is using (or even can use) it with recent Heat. - The repository still contains the utility for CloudWatch API which was already removed from heat - The heat-cfn command relies on Keystone v2 instead of v3 - No release has been made since the repository was created Based on the current situation I'd propose retiring the repository. I'd appreciate any feedback and if I hear no objections then I'll start the official process to retire the repository. Thank you, Takashi

2 3

[kolla-ansible][Antelope][Senlin] Senlin Rolling Updates Not Working
by Devin Patterson 16 Jan '24

16 Jan '24

Hello, We have deployed Senlin clusters to manage VM health. When attempting to do rolling updates of VMs by updating the profile, the VMs go into an error state with the following error in the senlin engine logs. I've included the Heat template and OS commands. Any help would be appreciated. openstack cluster update --profile <new_profile> <senlin_cluster> cluster event show <event_d> status_reason | Failed in updating nodes. heat_template_version: 2021-04-16 description: OpenStack Heat template to deploy a Senlin autoscaling group resources: profile1: type: OS::Senlin::Profile properties: type: os.nova.server-1.0 properties: flavor: m1.small image: cirros-0.5.1 key_name: keypair networks: - network: network user_data: | #!/bin/sh echo 'hello, world' > /tmp/test_file batch_policy: type: OS::Senlin::Policy properties: type: senlin.policy.batch-1.0 bindings: - cluster: {get_resource: cluster} properties: min_in_service: 1 max_batch_size: 2 pause_time: 3 cluster: type: OS::Senlin::Cluster properties: desired_capacity: 8 min_size: 2 profile: { get_resource: profile1 } receiver_scale_out: type: OS::Senlin::Receiver properties: cluster: {get_resource: cluster} action: CLUSTER_SCALE_OUT type: webhook receiver_scale_in: type: OS::Senlin::Receiver properties: cluster: {get_resource: cluster} action: CLUSTER_SCALE_IN type: webhook outputs: receiver_scale_out: description: Webhook to scale out cluster. value: str_replace: template: curl -X POST LINK params: LINK: {get_attr: [receiver_scale_out, channel, alarm_url]} receiver_scale_in: description: Webhook to scale in cluster. value: str_replace: template: curl -X POST LINK params: LINK: {get_attr: [receiver_scale_in, channel, alarm_url]} Regards, Devin

1 0

[kolla-ansible][Antelope][Heat] Heat Autoscaling Scaling Policy URL
by Devin Patterson 16 Jan '24

16 Jan '24

Hello, I am attempting to create an auto-scaling group with Heat. The group is deployed properly along with scale up and scale down policies. When attempting to curl with POST, I get the following response. This is using the "alarm_url" from the scaling policy. Below is also the Heat template being utilized. Below the error response is the curl command. Wondering why there is an AWSAccessKeyId in the signature of the URL as well. <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>500 Internal Server Error</title> </head><body> <h1>Internal Server Error</h1> <p>The server encountered an internal error or misconfiguration and was unable to complete your request.</p> <p>Please contact the server administrator at [no address given] to inform them of the time this error occurred, and the actions you performed just before this error.</p> <p>More information about this error may be available in the server error log.</p> </body></html> CURL COMMAND: curl -X POST <external-os-url>/v1/signal/arn%3Aopenstack%3Aheat%3A%omitted3Astacks/stack_1/omitted/resources/scale_down_policy\?SignatureMethod\=HmacSHA256\&SignatureVersion\=2\&AWSAccessKeyId\=omitted\&Signature\=omitted TEMPLATE: heat_template_version: 2021-04-16 description: OpenStack Heat template to deploy a autoscaling group resources: asg: type: OS::Heat::AutoScalingGroup properties: desired_capacity: 6 max_size: 8 min_size: 2 rolling_updates: min_in_service: 1 max_batch_size: 2 pause_time: 3 resource: type: OS::Nova::Server properties: flavor: m1.small image: image key_name: my-keypair networks: - network: network user_data: | #!/bin/sh echo 'hello, world123' > /tmp/test_file scale_up_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: { get_resource: asg } cooldown: 60 scaling_adjustment: 1 scale_down_policy: type: OS::Heat::ScalingPolicy properties: adjustment_type: change_in_capacity auto_scaling_group_id: { get_resource: asg } cooldown: 60 scaling_adjustment: -1 outputs: scale_up_url: description: > This URL is the webhook to scale up the group. You can invoke the scale-up operation by doing an HTTP POST to this URL; no body nor extra headers are needed. value: {get_attr: [scale_up_policy, alarm_url]} scale_dn_url: description: > This URL is the webhook to scale down the group. You can invoke the scale-down operation by doing an HTTP POST to this URL; no body nor extra headers are needed. value: {get_attr: [scale_down_policy, alarm_url]} asg_size: description: > This is the current size of the auto scaling group. value: {get_attr: [asg, current_size]} server_list: description: > This is a list of server names that are part of the group. value: {get_attr: [asg, outputs_list, name]} networks: description: > This is a map of server resources and their networks. value: {get_attr: [asg, outputs, networks]} server_ips: description: > This is a list of first ip addresses of the servers in the group for a specified network. value: {get_attr: [asg, outputs_list, networks, {get_param: network}, 0]} Regards, Devin

1 0

[magnum] k8s-keystone-auth pod CrashLoopBackOff
by Eugen Block 16 Jan '24

16 Jan '24

Hi there, I've noticed several magnum related threads and thought I'd give it a try while I keep digging. I've only recently started to work with magnum, and this is how far I get. Note that this openstack is still running on version Victoria, I'm working towards migrating it to a newer release. My coe cluster template currently looks like this (I'll omit the defaults): | labels | {'kube_tag': 'v1.18.2', 'hyperkube_prefix': 'k8s.gcr.io/'} | | https_proxy | http://<IP>:<PORT> | | http_proxy | http://<IP>:<PORT> | | image_id | fedora-coreos33 | | dns_nameserver | <MY_DNS> | I've already tried quite a lot of different kube_tags and hyperkube_prefixes, this combination gets me at least to have some running containers: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# podman ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 7fbe3f8f308f docker.io/openstackmagnum/heat-container-agent:victoria-stable-1 /usr/bin/start-he... 42 minutes ago Up 42 minutes ago heat-container-agent a8b6ae357034 k8s.gcr.io/hyperkube:v1.18.2 kube-apiserver --... 40 minutes ago Up 40 minutes ago kube-apiserver cb7774fbf849 k8s.gcr.io/hyperkube:v1.18.2 kube-controller-m... 40 minutes ago Up 40 minutes ago kube-controller-manager 448f1c02cbd4 k8s.gcr.io/hyperkube:v1.18.2 kube-scheduler --... 40 minutes ago Up 40 minutes ago kube-scheduler 4be28a0a2271 k8s.gcr.io/hyperkube:v1.18.2 kubelet --logtost... 40 minutes ago Up 40 minutes ago kubelet 542ffdb081e7 k8s.gcr.io/hyperkube:v1.18.2 kube-proxy --logt... 40 minutes ago Up 40 minutes ago kube-proxy 61874a4e4595 quay.io/coreos/etcd:v3.4.6 /usr/local/bin/et... 40 minutes ago Up 40 minutes ago etcd But not all pods are starting: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl get pods -o wide --kubeconfig=/etc/kubernetes/admin.conf -A NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES kube-system coredns-786ffb7797-dpkbx 1/1 Running 0 25m 10.100.0.3 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system coredns-786ffb7797-mslfk 1/1 Running 0 25m 10.100.0.2 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system dashboard-metrics-scraper-6b4884c9d5-nrvbg 1/1 Running 0 25m 10.100.0.5 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system k8s-keystone-auth-qt7n5 0/1 CrashLoopBackOff 4 5m12s 10.0.0.116 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system kube-dns-autoscaler-75859754fd-69zxx 0/1 Pending 0 25m <none> <none> <none> <none> kube-system kube-flannel-ds-cn5jf 1/1 Running 0 25m 10.0.0.116 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system kubernetes-dashboard-c98496485-qw4hl 1/1 Running 0 25m 10.100.0.4 kubernetes-cluster20-buuxqog6cfnn-master-0 <none> <none> kube-system magnum-metrics-server-79556d6999-hkqcw 0/1 Pending 0 25m <none> <none> <none> <none> I didn't look too deep into the two other failing pods, I think the keystone pod is more important at this point or maybe even is required for the other pods to start, I don't know yet. I can pull the image manually and the "describe pod" command also shows that the container was pulled successfully, but I can't find other hints what is actually failing. Here is the full output of "describe pod": ---snip--- [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system describe pod/k8s-keystone-auth-qt7n5 Name: k8s-keystone-auth-qt7n5 Namespace: kube-system Priority: 0 Node: kubernetes-cluster20-buuxqog6cfnn-master-0/10.0.0.116 Start Time: Mon, 15 Jan 2024 09:13:16 +0000 Labels: controller-revision-hash=6875f96b46 k8s-app=k8s-keystone-auth pod-template-generation=1 Annotations: kubernetes.io/psp: magnum.privileged Status: Running IP: 10.0.0.116 IPs: IP: 10.0.0.116 Controlled By: DaemonSet/k8s-keystone-auth Containers: k8s-keystone-auth: Container ID: docker://adca0b9a9eb011f22530886a42bf862b4cfa7e6b7965cd77d6dac4f37222b827 Image: docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0 Image ID: docker-pullable://k8scloudprovider/k8s-keystone-auth@sha256:ab72a7e5b9eca9af2762796690be800bce2de25dee8bd5ad55d0996b29c85146 Port: 8443/TCP Host Port: 8443/TCP Args: ./bin/k8s-keystone-auth --tls-cert-file /etc/kubernetes/certs/server.crt --tls-private-key-file /etc/kubernetes/certs/server.key --policy-configmap-name k8s-keystone-auth-policy --keystone-url http://<CONTROLLER>:5000/v3 --sync-configmap-name keystone-sync-policy --keystone-ca-file /etc/kubernetes/ca-bundle.crt --listen 127.0.0.1:8443 State: Terminated Reason: Error Exit Code: 1 Started: Mon, 15 Jan 2024 09:23:52 +0000 Finished: Mon, 15 Jan 2024 09:23:52 +0000 Last State: Terminated Reason: Error Exit Code: 1 Started: Mon, 15 Jan 2024 09:18:41 +0000 Finished: Mon, 15 Jan 2024 09:18:41 +0000 Ready: False Restart Count: 6 Requests: cpu: 200m Environment: <none> Mounts: /etc/kubernetes from ca-certs (ro) /etc/kubernetes/certs from k8s-certs (ro) /var/run/secrets/kubernetes.io/serviceaccount from k8s-keystone-auth-token-vhm4r (ro) Conditions: Type Status Initialized True Ready False ContainersReady False PodScheduled True Volumes: k8s-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes/certs HostPathType: DirectoryOrCreate ca-certs: Type: HostPath (bare host directory volume) Path: /etc/kubernetes HostPathType: DirectoryOrCreate k8s-keystone-auth-token-vhm4r: Type: Secret (a volume populated by a Secret) SecretName: k8s-keystone-auth-token-vhm4r Optional: false QoS Class: Burstable Node-Selectors: node-role.kubernetes.io/master= Tolerations: :NoSchedule :NoExecute CriticalAddonsOnly node.kubernetes.io/disk-pressure:NoSchedule node.kubernetes.io/memory-pressure:NoSchedule node.kubernetes.io/network-unavailable:NoSchedule node.kubernetes.io/not-ready:NoExecute node.kubernetes.io/pid-pressure:NoSchedule node.kubernetes.io/unreachable:NoExecute node.kubernetes.io/unschedulable:NoSchedule Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Scheduled <unknown> default-scheduler Successfully assigned kube-system/k8s-keystone-auth-qt7n5 to kubernetes-cluster20-buuxqog6cfnn-master-0 Warning Failed 9m52s kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Error: ErrImagePull Warning Failed 9m52s kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Failed to pull image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0": rpc error: code = Unknown desc = Get "https://registry-1.docker.io/v2/k8scloudprovider/k8s-keystone-auth/manifest…": Internal Server Error Normal Pulling 9m37s (x5 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Pulling image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0" Normal Pulled 9m36s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Successfully pulled image "docker.io/k8scloudprovider/k8s-keystone-auth:v1.18.0" Normal Created 9m35s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Created container k8s-keystone-auth Normal Started 9m35s (x4 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Started container k8s-keystone-auth Warning BackOff 33s (x48 over 10m) kubelet, kubernetes-cluster20-buuxqog6cfnn-master-0 Back-off restarting failed container ---snip--- I'm obviously not allowed to get logs from the pod: [root@kubernetes-cluster20-buuxqog6cfnn-master-0 ~]# /var/srv/magnum/bin/kubectl --kubeconfig=/etc/kubernetes/admin.conf -n kube-system logs pod/k8s-keystone-auth-qt7n5 Error from server: Get https://10.0.0.116:10250/containerLogs/kube-system/k8s-keystone-auth-qt7n5/…: Forbidden I appreciate any hints! Thanks! Eugen

1 3

[keystone] Keystone as a Service Provider, 401 - Unauthorized Exception for SSO
by joeldag7＠gmail.com 16 Jan '24

16 Jan '24

We're currently working on implementing SSO for our university's OpenStack environment using OpenStack Ansible (Yoga version). Our goal is to allow students to log in with their university credentials via an Identity Provider - currently we're using our own Keycloak instance as IdP. The plan is to automatically create a new project for first time users, while returning students will be redirected to their existing projects to manage their instances. For initial testing, we're directing all users to a single project. Eventually, we would like to script the dynamic creation of projects on the controller node for first-time logins later. However, we've run into some problems: Our Keystone configuration in the user_variables.yml file is as follows: ``` keystone_sp: apache_mod: 'mod_auth_openidc' cadf_notifications: true trusted_dashboard_list: - "https://openstack-example.com/auth/websso/" - "http://example.keycloak.com/realms/openstack*" - "https://openstack-example.com/auth/login/" trusted_idp_list: - name: "oidc-idp" oidc_provider_metadata_url: http://example.keycloak.com/realms/openstack/.well-known/openid-configurati… oidc_client_id: ******** oidc_client_secret: ********* oidc_crypto_passphrase: ****** oidc_redirect_uri: https://openstack-example.com/v3/OS-FEDERATION/identity_providers/oidc-idp/… entity_ids: - 'http://exmaple.keycloak.com/realms/openstack' federated_identities: - domain: default project: fedproject group: fedgroup role: member protocols: - name: openid mapping: name: openid-mapping rules: - remote: - type: email local: - group: name: fedgroup domain: name: Default user: name: '{0}' ``` This setup is based on the openstack documentation (https://docs.openstack.org/openstack-ansible-os_keystone/yoga/configure-fed…) After many attempts with different configurations, this one successfully redirects to our Keycloak IdP for authentication. However, after login, when redirected back to OpenStack, we get a 401 Unauthorised error stating "The request you have made requires authentication". The SAML tracer logs confirm that Keycloak issues a valid token along with the user information. Despite this, we find no additional error details in the Keystone container, Apache2 and controller logs. This leads me to believe that the problem may lie in the mapping configuration, although it appears to work when tested with the keystone-manage mapping-engine tool. But still, the doc (https://docs.openstack.org/keystone/queens/advanced-topics/federation/mappi…) claims that a 401 exception could be caused by the mapping. Any insight or suggestions on how to solve this problem would be greatly appreciated, particularly regarding the mapping process during authentication. I'm also looking for advice on how to extract more detailed error information. At the moment, the logs don't reveal much beyond the 401 error message. Am I missing something or is this all the information OpenStack typically provides in such cases? Thank you for your help and guidance.

1 0

OpenStack D Release Name
by Allison Price 15 Jan '24

15 Jan '24

Hi everyone, I am excited to share that the name for the OpenStack D release, scheduled for 2H 2024 is Dalmatian. (Congrats, Amy!) We look forward to developing the 30th release of OpenStack! Cheers, Allison

4 3

[tc] Next Technical Committee Meeting Jan 16 2024, 1800 UTC
by Jay Faulkner 15 Jan '24

15 Jan '24

Hi all, The next weekly OpenStack Technical Committee meeting is tomorrow, Tuesday Jan 16 at 1800 UTC in #openstack-tc on OFTC. The proposed agenda is: Agenda - Roll call - Follow up on tracked action items - JayF - Email mailing list a summary of the discussion around Monasca, highlighting issues around trust and project activity along with a link to the meeting log. - slaweq - Reach out privately to volunteer to maintain Monasca about official vs unofficial options - gmann - Reach out privately to volunteer to maintain Senlin about next steps - Gate health check - Implementation of Unmaintained branch statuses - Inactive projects for Caracal - It is past C-2, these projects will not be in Caracal release. Would we like to take further action in Caracal? - Monasca - Senlin - Freezer - Solum - EC2 API - Sahara - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Open Discussion and Reviews Thanks, Jay Faulkner

1 0

[neutron] Bug deputy - week of Jan 8
by Sławek Kapłoński 15 Jan '24

15 Jan '24

Hi, It was pretty quiet week. Below is summary of the bugs opened this last week and its current status. ## High * https://bugs.launchpad.net/neutron/+bug/2048745 - [ovn] FIP not working when mixing vlan and geneve tenant networks - not assigned, many things to consider before we will start working on this improvement really * https://bugs.launchpad.net/neutron/+bug/2048785 - Trunk parent port (tpt port) vlan_mode is wrong in ovs - WIP fix already started by Bence: https://review.opendev.org/c/openstack/neutron/+/905125 * https://bugs.launchpad.net/neutron/+bug/2049265 - [OVN] The ``MaintenanceWorker`` shoudl register the "Private_Chassis" table in the SB connection - assigned to Terry Wilson, fixes already proposed https://review.opendev.org/q/I711996b7644e80bc195833e4429e4d745728f9cf ## Medium * https://bugs.launchpad.net/neutron/+bug/2048979 - [ml2][ovs] ports without local vlan tag are processed on openflow security group - fix proposed by LIU: https://review.opendev.org/c/openstack/neutron/+/905289 ## Others * https://bugs.launchpad.net/neutron/+bug/2048773 - neutron-meter-agent do not support ovn driver - unassigned, it isn't really completly new RFE IMO but probably approach to take to close this gap may be discussed e.g. in the drivers meeting, * https://bugs.launchpad.net/neutron/+bug/2049398 - [Tempest] neutron_tempest_plugin.api.admin.test_external_network_extension.ExternalNetworksRBACTestJSON.test_delete_policies_while_tenant_attached_to_net "Created Network is Not an External Network" Error - needs some more info to be triaged, I will take care of it -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[Image Encryptione] Cancelling tomorrows meeting
by Josephine Seifert 14 Jan '24

14 Jan '24

Hi, unfortunately I have to cancel tomorrows meeting. greetings, Luzi (Josephine)

1 0

[kolla]Boot one VM with two GPU(in same numa)by pci passthrough cannot have GPUDirect P2P capability
by dazhaoyu＠126.com 12 Jan '24

12 Jan '24

I have two GPU cards, all of them was connect with one same numa CPU socket as below link info: https://paste.opendev.org/show/b7Qi8qCnbLVxO2W0JdQw/ I can boot one nova instance successfully with the two GPU cards by PCI Passthrough way. but in the booted instances, use deviceQuery method would get the below message: Peer access from NVIDIA RTX 6000(GPU0) -> NVIDIA RTX 6000(GPU1): NO Peer access from NVIDIA RTX 6000(GPU1) -> NVIDIA RTX 6000(GPU0): NO The expected return should be as below： Peer access from NVIDIA RTX 6000(GPU0) -> NVIDIA RTX 6000(GPU1): YES Peer access from NVIDIA RTX 6000(GPU1) -> NVIDIA RTX 6000(GPU0): YES so that the memory can be shared between the two GPUs. I'm running Openstack Xena release in Intel Xeon Gold 5220R CPU

1 0

[release] Release countdown for week R-11, Jan 15-19
by Elõd Illés 12 Jan '24

12 Jan '24

Development Focus ----------------- We are now past the Caracal-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (February 22nd, 2024). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (February 29th, 2024) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, February 29th, 2024. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (March 14th, 2024) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (March 28th, 2024) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: February 22nd, 2024 (R-6 week) Client library freeze: February 29th, 2024 (R-5 week) Caracal-3 milestone: February 29th, 2024 (R-5 week) 2024.1 Caracal final release: April 3rd, 2024 2024.2 'D' virtual PTG ( https://openinfra.dev/ptg/ ): April 8th-12th, 2024 Előd Illés irc: elodilles @ #openstack-release

1 0

[openstack][keystone] user who mapped from openid cannot create application credentials
by Nguyễn Hữu Khôi 12 Jan '24

12 Jan '24

Hello. I have a problem that I use openid for authentication to my cloud but my users cannot create application credentials. I must add them to a group via Horizon UI to use application credentials. Is there any way to do that without adding users to a group from Horizon? Thank you. Nguyen Huu Khoi

1 1

[kolla-ansible][magnum]Private CA in Fedora-CoreOS
by Franck VEDEL 12 Jan '24

12 Jan '24

Hello, I have two platforms: on the first (Yoga), which is connected with public names and addresses, certificates issued by trusted CAs, no problem, creating a cluster template and deploying a cluster based on Fedora-CoreOS-35 works perfectly. On the second, I have a private certification authority (and privates addresses) and when I create a template then deploy a cluster, obviously there is a problem (in each node of the cluster) with the certificate issued by an untrusted CA. For the cluster to be deployed, I need to connect to the nodes, copy my CA certificate to /etc/pki/ca-trust/source/anchors/ then run the "update-ca-trust" command. My question is: is it possible to deploy a cluster using this method but without having to do what I am obliged to do if the CA is not trusted? If the answer exists, I couldn't find it in the documentation, I apologize. Thanks in advance Franck

1 0

[masakari] is going to be deprecated?!
by garcetto 12 Jan '24

12 Jan '24

good morning, maybe my comprehension fault, but is masakari going to be abandoned?! ... https://docs.openstack.org/kolla-ansible/latest/reference/compute/masakari-… "Support for deploying Masakari has been deprecated in the 2023.2 (Bobcat) release due to failures in the CI and lack of contributors working on fixing this."

2 2

[octavia] [octavia-ingress-controller][k8s]
by Johnny Yang 12 Jan '24

12 Jan '24

Hi, We are following https://github.com/kubernetes/cloud-provider-openstack/blob/master/docs/oct… to create octavia ingress controller. we are in openstack yoga version, and octavia ingress control version as ( registry.k8s.io/provider-os/octavia-ingress-controller:v1.28.1) we are getting the following error from octavia ingress controller pod =========== ❯ k logs octavia-ingress-controller-0 -n kube-system 2024/01/10 17:47:16 Running command: Command env: (log-file=, also-stdout=false, redirect-stderr=true) Run from directory: Executable path: /bin/octavia-ingress-controller Args (comma-delimited): /bin/octavia-ingress-controller,--config=/etc/config/octavia-ingress-controller-config.yaml 2024/01/10 17:47:16 Now listening for interrupts INFO [2024-01-10T17:47:16Z] Using config file file=/etc/config/octavia-ingress-controller-config.yaml W0110 17:47:16.597288 12 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work. FATAL [2024-01-10T17:47:16Z] failed to initialize openstack client error="Get \"/\": unsupported protocol scheme \"\"" 2024/01/10 17:47:16 running command: exit status 1 ======= the following are our service account/configmap/deployment yaml files: ============== serviceaccount.yaml ----- kind: ServiceAccount apiVersion: v1 metadata: name: octavia-ingress-controller namespace: kube-system --- kind: ClusterRoleBinding apiVersion: rbac.authorization.k8s.io/v1 metadata: name: octavia-ingress-controller roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: cluster-admin subjects: - kind: ServiceAccount name: octavia-ingress-controller namespace: kube-system --- config.yaml ---- kind: ConfigMap apiVersion: v1 metadata: name: octavia-ingress-controller-config namespace: kube-system data: config: | cluster-name: cedev17 openstack: auth_url: https://srelab501.wpc.az1.eng.pdx.wd:5000 project_domain_name: Default user_domain_name: Default project_name: cedev17.t501.eng.pdx.wd project_id: 94ba42c68e1346189b666f17e49e22f5 username: admin user-id: b0f2b611d99e444cbe1c1fa068940411 password: password region_name: RegionOne cacert: /etc/pki/tls/certs/ca-bundle.crt octavia: subnet-id: 37abc0d8-f5ab-4109-8864-622ab4b47b1f floating-network-id: 42d4ba58-ccd6-407b-a887-5727ee7fe275 manage-security-groups: false provider: amphora ----- --- kind: StatefulSet apiVersion: apps/v1 metadata: name: octavia-ingress-controller namespace: kube-system labels: k8s-app: octavia-ingress-controller spec: replicas: 1 selector: matchLabels: k8s-app: octavia-ingress-controller serviceName: octavia-ingress-controller template: metadata: labels: k8s-app: octavia-ingress-controller spec: serviceAccountName: octavia-ingress-controller tolerations: - effect: NoSchedule # Make sure the pod can be scheduled on master kubelet. operator: Exists - key: CriticalAddonsOnly # Mark the pod as a critical add-on for rescheduling. operator: Exists - effect: NoExecute operator: Exists imagePullSecrets: - name: regcred containers: - name: octavia-ingress-controller image: docker-dev-artifactory.workday.com/wpc5/dev/octavia-ingress-controller:v1.2… imagePullPolicy: IfNotPresent args: - /bin/octavia-ingress-controller - --config=/etc/config/octavia-ingress-controller-config.yaml volumeMounts: - mountPath: /etc/kubernetes name: kubernetes-config readOnly: true - name: ingress-config mountPath: /etc/config hostNetwork: true volumes: - name: kubernetes-config hostPath: path: /etc/kubernetes type: Directory - name: ingress-config configMap: name: octavia-ingress-controller-config items: - key: config path: octavia-ingress-controller-config.yaml ------ Thank you in advance for help! -Johnny

2 1

beginning with [masakari]
by Shubham Kumar Yadav 11 Jan '24

11 Jan '24

Hi, i have recently started working on Masakari (kubernetes in openstack) so i wanted some help with it please i was trying to test the masakari-instance-monitor & masakari-host-monitor but having trouble with it. i have created a pacemaker remote cluster (with pacemaker & corosync on controller nodes and pacemaker remote on compute node). *pcs statusCluster name: lab-clusterCluster Summary: * Stack: corosync * Current DC: lab-11053-26006-ceph-2 (version 2.1.2-ada5c3b36e2) - partition with quorum * Last updated: Thu Dec 14 08:41:41 2023 * Last change: Thu Dec 14 08:41:22 2023 by root via cibadmin on lab-11053-26006-ceph-1 * 6 nodes configured * 3 resource instances configuredNode List: * Online: [ lab-11053-26006-ceph-1 lab-11053-26006-ceph-2 lab-11053-26006-ceph-3 ] * RemoteOnline: [ lab-11053-26006-comp-1.cluster.local lab-11053-26006-comp-2.cluster.local lab-11053-26006-comp-3.cluster.local ]Full List of Resources: * lab-11053-26006-comp-1.cluster.local (ocf:pacemaker:remote): Started lab-11053-26006-ceph-1 * lab-11053-26006-comp-2.cluster.local (ocf:pacemaker:remote): Started lab-11053-26006-ceph-2 * lab-11053-26006-comp-3.cluster.local (ocf:pacemaker:remote): Started lab-11053-26006-ceph-3Daemon Status: corosync: active/enabled pacemaker: active/enabled pcsd: active/enabled* i have also created segment and added segment hosts(computes) in it. when i tried to test the masakari host monitor by manually powering off the compute node, masakari should move the vms from powered off computes to some running compute but after the compute is powered off vms still remains on the powered off compute. Am i missing some conf? is there any need to add some conf in nova and keystone too? *in masakari engine logs i seehost_failure.evacuate_all_instances = Trueinstance_failure.process_all_instances = Falsehost_failure.add_reserved_host_to_aggregate = Falsehost_failure.ha_enabled_instance_metadata_key = HA_Enabled log_opt_valueshost_failure.ignore_instances_in_error_state = Falsehost_failure.service_disable_reason = Masakari detected host failed.* *2 - question do i have to create some notification and vmoves?* openstack notification create <type> <hostname> <generated_time> <payload> if yes what is the use of notification & vmoves i ahve attached a file with the email which contains the componenets i created for masakari(segments &hosts in segments)

2 1

[senlin] Rolling updates issues future support and alternatives
by Devin Patterson 11 Jan '24

11 Jan '24

Hello folks, We are attempting to find a solution for tenants in our OpenStack environment and have been testing Senlin for auto-scaling, auto-healing and rolling updates. The goal is similar functionality to AWS auto-scaling groups. Auto-scaling and health check functionality is working properly but we are getting some errors when attempting to update Senlin clusters to new profiles. Specifically, we would like to update VM user_data and VM images and roll those changes out in alignment with batch policies. The Senlin clusters attempt to update but fail with errors in the senlin_engine logs either related to lack of support for user_data updates or duplicate images when updating images in the profile. Proxy.rebuild_server() got multiple values for argument 'image' 2024-01-11 16:16:10.050 1009 ERROR senlin.profiles.base [] The following properties are not updatable: user_data. 16:10.086 1009 ERROR senlin.engine.event [] NODE_UPDATE[] error: Node update failed. Is anyone able to help with a path forward as well as provide any input on future support for Senlin as I saw some messages related to it no longer being supported? If so, can anyone recommend OpenStack project alternatives to achieve autoscaling, auto-healing and rolling updates for tenant VMs? Thanks in advance for any help, Devin

1 0

[magnum] autoscaling feature not working
by Satish Patel 11 Jan '24

11 Jan '24

Folks, I am trying to set the auto scaling feature in magnum "auto_scaling_enabled '' true but somehow the auto scaling is not working. truly speaking I don't understand the workflow of scaling up and down of worker nodes. How does magnum know that it requires scaling up and down without monitoring? On what basis will it trigger that action?

2 8

[Openstack-Chef][all][tc] Retiring Openstack-Chef
by Ghanshyam Mann 10 Jan '24

10 Jan '24

Hello Everyone, You might have seen that OpenStack-Chef project (provides the Chef cookbooks for OpenStack life cycle management) is leaderless in this cycle, and further investigation by TC found that this project's health is not green and does not have maintainers. - https://etherpad.opendev.org/p/2024.1-leaderless A few of the stats about this project's activities: - gate health unclear, no activity for > 1y in openstack/openstack-chef repo - most recent branch is stable/train - zuul config errors fixed by frickler We contacted the previous PTL Lance Albertson and he would not be able to continue maintaining it and are ok with the retirement. Considering the current situation and no maintainers in this project, TC is planning to retire this project. This is the last call if any of you or your organization are interested in maintaining it (or using it and would like to discuss the possible path forward). Feel free to reply to this email or add your feedback in Gerrit: - https://review.opendev.org/c/openstack/governance/+/905279 -gmann

1 0

[all] Nomination for extra-ACs
by Jay Faulkner 10 Jan '24

10 Jan '24

OpenStackers: Elections are beginning soon. This means as a community, we have a responsibility to identify and credit any extra Active Contributors (extra ACs) who may not have committed code in the last six months. Being named an extra AC has a simple, but important impact: it permits the individual named to vote in elections for the Technical Committee. Additionally, it's an important form of recognition which can encourage future non-code contributions. If there's someone who has made non-code contributions to a project who should be recognized, please contact me, in my role as the Technical Committee Chair (per the TC Charter https://governance.openstack.org/tc/reference/charter.html#voters-for-tc-se…) and the TC will evaluate the request and approve it if reasonable. Please note; this is not just an exercise for PTLs. Self-nominating for extra-AC status is not only permitted; it's encouraged -- if you've contributed, we want to recognize you. Additionally, any existing project contributors are eligible to nominate peers they think are deserving. Thank you in advance for taking time to recognize your upstream peers! Jay Faulkner OpenStack TC Chair

1 0