[openstack][kolla-ansible] kolla-ansible stucked when reconfigure
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
Hi Could you please share debug log when running kolla-ansible and docker log in stucked host
In Vietnameses Bạn có thể chia sẻ debug của kolla-ansible và log của docker ở host bị treo ko?
On Fri, Dec 22, 2023 at 2:54 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
Hello,
It just likes that
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/ee5fa6fb8b"' -tt compute15 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1703247096.67589-2724582-15706725226175/AnsiballZ_setup.py && sleep 0'"'"''
Pending forever.
Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 5:22 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
Hi Could you please share debug log when running kolla-ansible and docker log in stucked host
In Vietnameses Bạn có thể chia sẻ debug của kolla-ansible và log của docker ở host bị treo ko?
On Fri, Dec 22, 2023 at 2:54 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello guys.
I am operating an openstack cluster with kolla but sometimes I cannot reconfigure. It stucked at some hosts(gather fact) and I need to restart docker service on the stucked host to continue. I tried to google but I cannot find the root cause.
Would you share some experiences on it because my solution causes instances on host to timeout some seconds.
Thank you.
Nguyen Huu Khoi
-- Thu.
-- Thu.
Hello,
I just issue this command with stuck hosts:
ansible -i inventoryTest all -m shell -a uptime
then everything is smooth
Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 7:13 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Hello,
It just likes that
SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o ConnectTimeout=10 -o 'ControlPath="/root/.ansible/cp/ee5fa6fb8b"' -tt compute15 '/bin/sh -c '"'"'/usr/bin/python3 /root/.ansible/tmp/ansible-tmp-1703247096.67589-2724582-15706725226175/AnsiballZ_setup.py && sleep 0'"'"''
Pending forever.
Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 5:22 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
Hi Could you please share debug log when running kolla-ansible and docker log in stucked host
In Vietnameses Bạn có thể chia sẻ debug của kolla-ansible và log của docker ở host bị treo ko?
On Fri, Dec 22, 2023 at 2:54 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Oh. I ran without -vvv. I use it only for detecting which stucked. Nguyen Huu Khoi
On Fri, Dec 22, 2023 at 2:44 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hêlo. What do you mean?
On Fri, Dec 22, 2023, 2:20 PM Hoai-Thu Vuong thuvh87@gmail.com wrote:
have you captured debug logs with -vvv ?
On Wed, Dec 20, 2023 at 5:28 PM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
Hello.
I dont know why but with some tests and verificaton. When reconfiguration is stucked. I ran ad hoc with a stuck host or all host: "ansible -i inventoryTest all -m shell -a uptime" then stuck was gone. I can reconfigure.
On Tue, Nov 21, 2023, 6:49 AM Nguyễn Hữu Khôi < nguyenhuukhoinw@gmail.com> wrote:
> Hello guys. > > I am operating an openstack cluster with kolla but sometimes I > cannot reconfigure. It stucked at some hosts(gather fact) and I need to > restart docker service on the stucked host to continue. I tried to google > but I cannot find the root cause. > > Would you share some experiences on it because my solution causes > instances on host to timeout some seconds. > > Thank you. > > Nguyen Huu Khoi >
-- Thu.
-- Thu.
participants (2)
-
Hoai-Thu Vuong
-
Nguyễn Hữu Khôi