kubernetes cluster stuck in CREATE_IN_PROGRESS but stack list shows CREATE_COMPLETE
Dear Team, I have configured Magnum and I am able to kubernetes clusters and when I log in to the nodes everything is OK. But Minus the first cluster I created which completed well as per the openstack coe cluster list command and the output on Horizon, the rest of the clusters get stack in CREATE_IN_PROGRESS . Please check the below commands output and advise. (kolla-openstack) stack@deployment:~$ openstack stack list +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | None | | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | None | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ (kolla-openstack) stack@deployment:~$ openstack coe cluster list +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | 2 | 1 | CREATE_IN_PROGRESS | None | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ Regards Tony Karera
Hi Tony, Good to see your cluster can be created "successfully" ;) Technically, Magnum doesn't maintain the cluster status, the status is synced from Heat. That said, if the stack is created successfully, but the magnum cluster is in progress, that means something wrong when syncing the status. The issue happened before, I just cannot remember the exact bug ID, you should be able to find it by google. And here is the code, https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L... you can just add some breakpoints to debug it, it shouldn't be hard. On 15/09/21 8:54 pm, Karera Tony wrote:
Dear Team,
I have configured Magnum and I am able to kubernetes clusters and when I log in to the nodes everything is OK.
But Minus the first cluster I created which completed well as per the openstack coe cluster list command and the output on Horizon, the rest of the clusters get stack in CREATE_IN_PROGRESS .
Please check the below commands output and advise.
(kolla-openstack) stack@deployment:~$ openstack stack list +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | None | | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | None | +--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ (kolla-openstack) stack@deployment:~$ openstack coe cluster list +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | 2 | 1 | CREATE_IN_PROGRESS | None | +--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ Regards
Tony Karera
-- Cheers & Best regards, ------------------------------------------------------------------------------ Feilong Wang (王飞龙) (he/him) Head of Research & Development Catalyst Cloud Aotearoa's own Mob: +64 21 0832 6348 | www.catalystcloud.nz Level 6, 150 Willis Street, Wellington 6011, New Zealand CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipient, any use, reliance upon, disclosure or copying of this email or its attachments is unauthorised. If you have received this email in error, please reply via email or call +64 21 0832 6348. ------------------------------------------------------------------------------
Hello Feilong, The Clusters are now Ok and they are all in CEATE_COMPLETE after restarting the heat and magnum containers. Thanks Regards Tony Karera On Wed, Sep 15, 2021 at 8:48 PM feilong <feilong@catalyst.net.nz> wrote:
Hi Tony,
Good to see your cluster can be created "successfully" ;) Technically, Magnum doesn't maintain the cluster status, the status is synced from Heat. That said, if the stack is created successfully, but the magnum cluster is in progress, that means something wrong when syncing the status. The issue happened before, I just cannot remember the exact bug ID, you should be able to find it by google. And here is the code, https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L... you can just add some breakpoints to debug it, it shouldn't be hard.
On 15/09/21 8:54 pm, Karera Tony wrote:
Dear Team,
I have configured Magnum and I am able to kubernetes clusters and when I log in to the nodes everything is OK.
But Minus the first cluster I created which completed well as per the openstack coe cluster list command and the output on Horizon, the rest of the clusters get stack in CREATE_IN_PROGRESS .
Please check the below commands output and advise.
(kolla-openstack) stack@deployment:~$ openstack stack list
+--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | None | | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | None |
+--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ (kolla-openstack) stack@deployment:~$ openstack coe cluster list
+--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status |
+--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | 2 | 1 | CREATE_IN_PROGRESS | None |
+--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ Regards
Tony Karera
-- Cheers & Best regards, ------------------------------------------------------------------------------ Feilong Wang (王飞龙) (he/him) Head of Research & Development
Catalyst Cloud Aotearoa's own
Mob: +64 21 0832 6348 | www.catalystcloud.nz Level 6, 150 Willis Street, Wellington 6011, New Zealand
CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipient, any use, reliance upon, disclosure or copying of this email or its attachments is unauthorised. If you have received this email in error, please reply via email or call +64 21 0832 6348. ------------------------------------------------------------------------------
Hello Tony, I am suspecting that you are hitting the same bug as mine. You can see full details of bug here. https://storyboard.openstack.org/#!/story/2009141 Usually what happens it magnum-conductor service got crash randomly. You can confirm this by executing the command: openstack coe service list You will see that conductor service is down, once you restart conductor service, the cluster state will be updated to CREATE_COMPLETE. Ammad On Thu, Sep 16, 2021 at 10:02 AM Karera Tony <tonykarera@gmail.com> wrote:
Hello Feilong,
The Clusters are now Ok and they are all in CEATE_COMPLETE after restarting the heat and magnum containers.
Thanks
Regards
Tony Karera
On Wed, Sep 15, 2021 at 8:48 PM feilong <feilong@catalyst.net.nz> wrote:
Hi Tony,
Good to see your cluster can be created "successfully" ;) Technically, Magnum doesn't maintain the cluster status, the status is synced from Heat. That said, if the stack is created successfully, but the magnum cluster is in progress, that means something wrong when syncing the status. The issue happened before, I just cannot remember the exact bug ID, you should be able to find it by google. And here is the code, https://github.com/openstack/magnum/blob/master/magnum/service/periodic.py#L... you can just add some breakpoints to debug it, it shouldn't be hard.
On 15/09/21 8:54 pm, Karera Tony wrote:
Dear Team,
I have configured Magnum and I am able to kubernetes clusters and when I log in to the nodes everything is OK.
But Minus the first cluster I created which completed well as per the openstack coe cluster list command and the output on Horizon, the rest of the clusters get stack in CREATE_IN_PROGRESS .
Please check the below commands output and advise.
(kolla-openstack) stack@deployment:~$ openstack stack list
+--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | ID | Stack Name | Project | Stack Status | Creation Time | Updated Time |
+--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ | e9bfad7c-c82e-4fc6-9f05-793b31631761 | k8s-cluster2-zsgfkrobjlw6 | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-15T08:01:50Z | None | | 9dde1665-8e50-4482-a986-810eb2a9cbaa | k8s-cluster1-eogcos7ee3ub | 456586218b26442ebeff03643684faed | CREATE_COMPLETE | 2021-09-10T07:27:41Z | None |
+--------------------------------------+---------------------------+----------------------------------+-----------------+----------------------+--------------+ (kolla-openstack) stack@deployment:~$ openstack coe cluster list
+--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status |
+--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ | 277e0047-4ae1-4006-869e-a824cc602bcc | k8s-cluster1 | Newkey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | 4336f9d9-9bea-477f-bc13-c16ff0f7e141 | k8s-cluster2 | Newkey | 2 | 1 | CREATE_IN_PROGRESS | None |
+--------------------------------------+--------------+---------+------------+--------------+--------------------+---------------+ Regards
Tony Karera
-- Cheers & Best regards, ------------------------------------------------------------------------------ Feilong Wang (王飞龙) (he/him) Head of Research & Development
Catalyst Cloud Aotearoa's own
Mob: +64 21 0832 6348 | www.catalystcloud.nz Level 6, 150 Willis Street, Wellington 6011, New Zealand
CONFIDENTIALITY NOTICE: This email is intended for the named recipients only. It may contain privileged, confidential or copyright information. If you are not the named recipient, any use, reliance upon, disclosure or copying of this email or its attachments is unauthorised. If you have received this email in error, please reply via email or call +64 21 0832 6348. ------------------------------------------------------------------------------
-- Regards, Syed Ammad Ali
participants (3)
-
Ammad Syed
-
feilong
-
Karera Tony