<div dir="ltr">Hi Yatin,<div><br></div><div>Thanks a lot for your help. I am deleting the stack and running the overcloud deploy command as a process.</div><div><br></div><div>Changing the NTP settings worked for me in proceeding ahead.</div><div><br></div><div>But it seems the issues are not ending here.</div><div><br></div><div>I would require some more help from you in order to deploy this.</div><div><br></div><div><b>Issue:</b></div><div><br></div>FATAL | Check Keystone service status | undercloud | item=heat-cfn | error={"ansible_job_id": "687227427425.307276", "ansible_loop_var": "tripleo_keystone_resources_service_async_result_item", "attempts": 1, "changed": false, "extra_data": {"data": null, "details": "The request you have made requires authentication.", "response": "{\"error\":{\"code\":401,\"message\":\"The request you have made requires authentication.\",\"title\":\"Unauthorized\"}}\n"}, "finished": 1, "msg": "Failed to list services: Client Error for url: <a href="http://10.10.30.222:5000/v3/services">http://10.10.30.222:5000/v3/services</a>, <b>The request you have made requires authentication.",</b> "tripleo_keystone_resources_service_async_result_item": {"ansible_job_id": "687227427425.307276", "ansible_loop_var": "tripleo_keystone_resources_data", "changed": true, "failed": false, "finished": 0, "results_file": "/root/.ansible_async/687227427425.307276", "started": 1, "tripleo_keystone_resources_data": {"key": "heat-cfn", "value": {"endpoints": {"admin": "<a href="http://10.10.30.222:8000/v1">http://10.10.30.222:8000/v1</a>", "internal": "<a href="http://10.10.30.222:8000/v1">http://10.10.30.222:8000/v1</a>", "public": "<a href="http://10.10.30.222:8000/v1">http://10.10.30.222:8000/v1</a>"}, "region": "regionOne", "service": "cloudformation", "users": {"heat-cfn": {"password": "3f3tHhxhna1CpRVPMjF7po49F"}}}}}}<br><div> </div><div><br></div><div>PFA the ansible.log file.</div><div><br></div><div>Thanks your help and Patience.</div><div><br></div><div>Regards</div><div>Anirudh Gupta</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 2:28 PM Yatin Karel <<a href="mailto:ykarel@redhat.com">ykarel@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Anirudh,</div><div><br></div><div>Not sure what can cause this issue, and also the shared log file is incomplete. So I believe you tried the command on the same overcloud deployment which was failing earlier(when docker-ha.yaml was not passed). If yes, to rule out if the issue is caused by an already deployed environment can delete the overcloud and then redeploy with correct environment files as used in the last run.</div><div><br></div><div>One reason for the password expiration that i found could be the Time is not in Sync on the overcloud nodes. So it would be good to check that as well and fix(by using correct NTP sources) before attempting redeployment.<br></div><div><br></div><div>Thanks and regards</div><div>Yatin Karel<br></div><div><br></div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 2:03 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Yatin & Team<div><br></div><div>Thanks for your response.</div><div><br></div><div>When I executed the command as below, the installation moved ahead and encountered another error.</div><div><br></div><div>openstack overcloud deploy --templates \<br> -r /home/stack/templates/roles_data.yaml \<br> -e /home/stack/templates/node-info.yaml \<br> -e environment.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml \<br> -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \<br> -e /home/stack/containers-prepare-parameter.yaml<br></div><div><br></div><div><b>Issue:</b></div><div>The error was: keystoneauth1.exceptions.http.Unauthorized: <b>The password is expired and needs to be changed for user</b>: 4f7d1dbf58574e64af9e359cb98ccbbc. (HTTP 401) (Request-ID: req-b29aa655-e3ec-4d4b-8ada-397f9a132582)<br></div><div><br></div><div>I am attaching the ansible.logs for your reference. It would be a great help if you could suggest some pointers to resolve this issue.</div><div><br></div><div>Regards</div><div>Anirudh Gupta</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 11:13 AM Yatin Karel <<a href="mailto:ykarel@redhat.com" target="_blank">ykarel@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Anirudh,</div><div><br></div><div>As said order is important here, docker-ha.yaml should be followed by podman.yaml, the parameters in environment files override the parameters from previous environment files passed and that would make deployment to use podman instead of docker. Name of the parameter to which makes this switch is "<span>ContainerCli</span>".</div><div><br></div><div><br></div><div>Thanks and regards</div><div>Yatin Karel<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 10:59 AM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>If this is a docker-ha issue, then that has also been tried.</div><div><br></div><div>Since this is Centos 8, there is no docker available. If I pass the docker-ha.yml, then it gives the following error</div><div><br></div><div>FATAL | Pull undercloud.ctlplane.localdomain:8787/tripleotraincentos8/centos-binary-cinder-volume:current-tripleo image | overcloud-controller-1 | error={"changed": true, "cmd": "docker pull undercloud.ctlplane.localdomain:8787/tripleotraincentos8/centos-binary-cinder-volume:current-tripleo", "delta": "0:00:00.005932", "end": "2021-12-27 12:42:33.927484", "msg": "non-zero return code", "rc": 127, "start": "2021-12-27 12:42:33.921552", "stderr": "/bin/sh: docker: command not found", "<b>stderr_lines": ["/bin/sh: docker: command not found"], "stdout": "", "stdout_lines": []}</b><br></div><div><b><br></b></div><div>Regards</div><div>Anirudh Gupta</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 10:26 AM Yatin Karel <<a href="mailto:ykarel@redhat.com" target="_blank">ykarel@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi Anirudh,</div><div><br></div><div>Sorry which timer? Timer adjustment is not needed for the issue you are seeing, if you mean overcloud deploy timeout then overcloud deploy provides the option to do so using --timeout option. The best option for now is to try docker-ha and podman in order as suggested earlier.</div><div><br></div><div><br></div><div>Thanks and Regards</div><div>Yatin Karel<br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 10:12 AM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Thanks Yatin for your response.</div><div><br></div><div>Please suggest how can this timer be increased or any other steps that needs to be followed to rectify this?</div><div><br></div><div>Regards</div><div>Anirudh Gupta </div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Dec 28, 2021 at 10:08 AM Yatin Karel <<a href="mailto:ykarel@redhat.com" target="_blank">ykarel@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Anirudh,<br><br><br>On Mon, Dec 27, 2021 at 9:39 PM Anirudh Gupta <<a href="mailto:anyrude10@gmail.com" target="_blank">anyrude10@gmail.com</a>> wrote:<br>><br>> Hi Team,<br>><br>> I am trying to deploy TripleO Train with 3 controller and 1 Compute.<br>> For overcloud images, I have a registry server at undercloud only.<br>><br>> I executed the following command to deploy overcloud<br>><br>> openstack overcloud deploy --templates \<br>> -r /home/stack/templates/roles_data.yaml \<br>> -e /home/stack/templates/node-info.yaml \<br>> -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml \<br>> -e /home/stack/containers-prepare-parameter.yaml<br>><br>> The command ran for around 1.5 hrs and initially stack got successfully created and after that for 45 mins, ansible tasks were getting executed. It then gave following error in overcloud-controller-0<br>><br>> 2021-12-27 11:12:27,507 p=181 u=mistral n=ansible | 2021-12-27 11:12:27.506838 | 525400b1-b522-2a06-ea9d-00000000356f | OK | Debug output for task: Start containers for step 2 | overcloud-novacompute-0 | result={<br>> "changed": false,<br>> "failed_when_result": false,<br>> "start_containers_outputs.stdout_lines | default([]) | union(start_containers_outputs.stderr_lines | default([]))": [<br>> "f206c31a781641313aa4a0499c62475efc335de6faea785cd4e855dc32ebb571",<br>> "",<br>> "Info: Loading facts",<br>> "Notice: Compiled catalog for overcloud-novacompute-0.localdomain in environment production in 0.05 seconds",<br>> "Info: Applying configuration version '1640604309'",<br>> "Notice: /Stage[main]/Tripleo::Profile::Base::Neutron::Ovn_metadata_agent_wrappers/Tripleo::Profile::Base::Neutron::Wrappers::Haproxy[ovn_metadata_haproxy_process_wrapper]/File[/var/lib/neutron/ovn_metadata_haproxy_wrapper]/ensure: defined content as '{md5}5bb050ca70c01981975efad9d8f81f2d'",<br>> "Info: Tripleo::Profile::Base::Neutron::Wrappers::Haproxy[ovn_metadata_haproxy_process_wrapper]: Unscheduling all events on Tripleo::Profile::Base::Neutron::Wrappers::Haproxy[ovn_metadata_haproxy_process_wrapper]",<br>> "Info: Creating state file /var/lib/puppet/state/state.yaml",<br>> "Notice: Applied catalog in 0.01 seconds",<br>> "Changes:",<br>> " Total: 1",<br>> "Events:",<br>> " Success: 1",<br>> "Resources:",<br>> " Changed: 1",<br>> " Out of sync: 1",<br>> " Skipped: 7",<br>> " Total: 8",<br>> "Time:",<br>> " File: 0.00",<br>> " Transaction evaluation: 0.01",<br>> " Catalog application: 0.01",<br>> " Config retrieval: 0.09",<br>> " Last run: 1640604309",<br>> " Total: 0.01",<br>> "Version:",<br>> " Config: 1640604309",<br>> " Puppet: 5.5.10",<br>> "Error executing ['podman', 'container', 'exists', 'nova_compute_init_log']: returned 1",<br>> "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_compute_init_log', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id",<br>> "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=nova_compute_init_log', '--format', '{{.Names}}']\"",<br>> "Error executing ['podman', 'container', 'exists', 'create_haproxy_wrapper']: returned 1",<br>> "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_haproxy_wrapper', '--filter', 'label=config_id=tripleo_step2', '--format', '{{.Names}}']\" - retrying without config_id",<br>> "Did not find container with \"['podman', 'ps', '-a', '--filter', 'label=container_name=create_haproxy_wrapper', '--format', '{{.Names}}']\""<br>> ]<br>> }<br><br>This is not the actual error, actual error is: puppet-user: Error: /Stage[main]/Tripleo::Profile::Base::Rabbitmq/Rabbitmq_policy[ha-all@/]: Could not evaluate: Command is still failing after 180 seconds expired!"<br><br>><br>> I am also attaching ansible.log file for more information.<br>><br>> Note: On Centos 8, there is no docker, so I didn't pass docker-ha.yml<br>For enabling HA and with podman in Train on CentOS8, you need to pass both docker-ha.yaml and podman.yaml in order(<b>order is important here</b>, so -e /usr/share/openstack-tripleo-heat-templates/environments/docker-ha.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/podman.yaml), this way you will have deployment with HA and podman, i agree docker-ha name is confusing here with podman but that has to be passed here to get the required deployment. Also with Ussuri+ HA is turned on by default so those releases may work even without passing docker-ha.yaml but for Train at least it's needed.<br>><br>> Can someone please help in resolving my issue<br><div>></div><div>As per your requirement I would suggest running with the above config.</div><div><br></div>> Regards<br><div>> Anirudh Gupta</div><div><br></div><div>Thanks and Regards</div><div>Yatin Karel<br></div></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>