[openstack][kolla-ansible] kolla-ansible stucked when reconfigure
Hello guys. I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause. Would you share some experiences on it because my solution causes instances on host to timeout some seconds. Thank you. Nguyen Huu Khoi
Hello. I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure. On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
have you captured debug logs with -vvv ? On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
Hêlo. What do you mean? On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
Hi Could you please share debug log when running kolla-ansible and docker log in stucked host In Vietnameses Bạn có thể chia sẻ debug của kolla-ansible và log của docker ở host bị treo ko? On Fri, Dec 22, 2023 at 2:54 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
-- Thu.
Hello, It just likes that SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/ee5fa6fb8b"' -tt compute15 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1703247096.67589-2724582-15706725226175/AnsiballZ_setup.py && sleep 0'"'"'' Pending forever. Nguyen Huu Khoi On Fri, Dec 22, 2023 at 5:22 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
Hi Could you please share debug log when running kolla-ansible and docker log in stucked host
In Vietnameses Bạn có thể chia sẻ debug của kolla-ansible và log của docker ở host bị treo ko?
On Fri, Dec 22, 2023 at 2:54 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
-- Thu.
Hello, I just issue this command with stuck hosts: ansible -i inventoryTest all -m shell -a uptime then everything is smooth Nguyen Huu Khoi On Fri, Dec 22, 2023 at 7:13 PM Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com> wrote:
Hello,
It just likes that
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/ee5fa6fb8b"' -tt compute15 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1703247096.67589-2724582-15706725226175/AnsiballZ_setup.py && sleep 0'"'"''
Pending forever.
Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 5:22 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
Hi Could you please share debug log when running kolla-ansible and docker log in stucked host
In Vietnameses Bạn có thể chia sẻ debug của kolla-ansible và log của docker ở host bị treo ko?
On Fri, Dec 22, 2023 at 2:54 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong <thuvh87@gmail.com> wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
> Hello guys. > > I am operating an openstack cluster with kolla but sometimes I > cannot reconfigure. It stucked at some hosts(gather fact) and I need to > restart docker service on the stucked host to continue. I tried to google > but I cannot find the root cause. > > Would you share some experiences on it because my solution causes > instances on host to timeout some seconds. > > Thank you. > > Nguyen Huu Khoi >
-- Thu.
-- Thu.
participants (2)
-
Hoai-Thu Vuong
-
Nguyễn Hữu Khôi