Re: about magnum capi for production
Little progress, now I am stuck here.
Error: admission webhook " validation.clusterresourceset.addons.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the
# openstack coe cluster show j5 +----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value
| +----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ | status | CREATE_FAILED
| | health_status | None
| | cluster_template_id | 2ddfdf2a-5962-42ee-a2e2-ff9e0b19dbe5
| | node_addresses | []
| | uuid | 96723c28-65d2-4add-a29a-cf47a78e32a2
| | stack_id | kube-a8dvb
| | status_reason | admission webhook " validation.clusterresourceset.addons.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the |
On Mon, Dec 18, 2023 at 9:53 AM Oliver Weinmann oliver.weinmann@me.com wrote:
Im focusing purely on kolla-ansible. For myself it would be very helpful to have all info in a single place, so I assume other would also benefit from it.
Von meinem iPhone gesendet
Am 18.12.2023 um 11:11 schrieb Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com
:
This is not about doc much. Because we have too many cloud environments. If we have a good blog, are you sure what you write will work well with all environments? Nguyen Huu Khoi
On Mon, Dec 18, 2023 at 5:01 PM Satish Patel satish.txt@gmail.com wrote:
Hi Oliver,
I am 100% with you, There isn't any good technical blog or document about integration with magnum-capi drivers. There is good information but not good technical details with simple steps to follow. How to troubleshoot components etc. for the last few days I am struggling and still have no luck. I have tried all possible combinations. If my setup works then I will surely write good blogs.
On Mon, Dec 18, 2023 at 3:33 AM Oliver Weinmann oliver.weinmann@me.com wrote:
Hi all,
I’m also trying to get the Vexxhost CAPI driver working under Kolla-Ansible. Many thanks to Nguyen Huu Khoi Github page. This was a very good starting point. My goal is to collect all the info to get it working in a single place (my blog). Currently the info is pretty much scattered on different websites, I managed to create the cluster template but the creation fails immediately. This is the error that I get in magnum-conductor.log:
==> /var/log/kolla/magnum/magnum-conductor.log <== 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/magnum/service/periodic.py", line 100, in update_status 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall ng.destroy() 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall return fn(self, *args, **kwargs) 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/magnum/objects/nodegroup.py", line 175, in destroy 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall self.dbapi.destroy_nodegroup(self.cluster_id, self.uuid) 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/magnum/db/sqlalchemy/api.py", line 832, in destroy_nodegroup 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall raise exception.NodeGroupNotFound(nodegroup=nodegroup_id) 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall magnum.common.exception.NodeGroupNotFound: Nodegroup 4277e9e6-5c3e-4cce-a1cf-1f5e8c2f0689 could not be found. 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall
(2023.1) [vagrant@seed ~]$ openstack coe cluster list
+--------------------------------------+--------------------------------------+---------+------------+--------------+-----------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status |
+--------------------------------------+--------------------------------------+---------+------------+--------------+-----------------+---------------+ | b4ce540f-78a9-4c5d-a687-e992b3bd19a7 | k8s-flan-small-37-v1.23.3-containerd | mykey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | e8acc6da-f937-4e8f-9df8-1728a8079ed0 | k8s-v1.24.16 | mykey | 2 | 1 | CREATE_FAILED | None |
+--------------------------------------+--------------------------------------+---------+------------+--------------+-----------------+---------------+ (2023.1) [vagrant@seed ~]$ openstack coe nodegroup list k8s-v1.24.16
+--------------------------------------+----------------+---------------------+--------------------------------------+------------+--------------------+--------+ | uuid | name | flavor_id | image_id | node_count | status | role |
+--------------------------------------+----------------+---------------------+--------------------------------------+------------+--------------------+--------+ | 21c10537-e3d3-44cf-8e58-731cfeb5b9fe | default-master | m1.kubernetes.small | 9d989f56-359b-4d6a-a914-926e0ea938d7 | 1 | CREATE_IN_PROGRESS | master | | 12dec017-38cc-42d9-b944-649ae356907d | default-worker | m1.kubernetes.small | 9d989f56-359b-4d6a-a914-926e0ea938d7 | 2 | CREATE_IN_PROGRESS | worker |
+--------------------------------------+----------------+---------------------+--------------------------------------+------------+--------------------+————+
Cheers, Oliver
[On 17. Dec 2023, at 22:43, kmceliker@gmail.com wrote:
Here is an example of a CAPI deployment code for OpenStack, mate - using the clusterctl tool and the cluster-template.yaml file. This code will create a cluster named capi-openstack with one control plane node and three worker nodes, using the ubuntu-2204 image and the m1.medium flavor. You need to replace the placeholders with your own values
# Install clusterctl curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.0.1/clus... -o clusterctl chmod +x ./clusterctl sudo mv ./clusterctl /usr/local/bin/clusterctl
# Set environment variables export OPENSTACK_CLOUD=<openstack-cloud> export OPENSTACK_USERNAME=<openstack-username> export OPENSTACK_PASSWORD=<openstack-password> export OPENSTACK_DOMAIN_NAME=<openstack-domain-name> export OPENSTACK_PROJECT_ID=<openstack-project-id> export OPENSTACK_SSH_KEY_NAME=<openstack-ssh-key-name> export OPENSTACK_DNS_NAMESERVERS=<openstack-dns-nameservers> export OPENSTACK_EXTERNAL_NETWORK_ID=<openstack-external-network-id> export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=m1.medium export OPENSTACK_NODE_MACHINE_FLAVOR=m1.medium export OPENSTACK_IMAGE_NAME=ubuntu-2204 export KUBERNETES_VERSION=v1.23.15
# Initialize clusterctl clusterctl init --infrastructure openstack
# Create cluster clusterctl config cluster capi-openstack --kubernetes-version $KUBERNETES_VERSION --control-plane-machine-count=1 --worker-machine-count=3 > cluster-template.yaml clusterctl create cluster --kubeconfig ~/.kube/config --infrastructure openstack:v0.6.0 --bootstrap kubeadm:v0.4.4 --control-plane kubeadm:v0.4.4 --cluster capi-openstack --namespace default --from cluster-template.yaml
Also you can turn on Enable to Magnum Report Log on OpenStack to provide with us or Take a look deep dively as the following link; https://github.com/kubernetes-sigs/cluster-api-provider-openstack
Best, Kerem Çeliker Head of Cloud Architecture tr.linkedin.com/in/keremceliker
Hello Oliver,
It is a very good article from your blog.
I think Openstack still need an OFFICIAL Forum.I talked about this some months ago. New users cannot search problems via email, a very hard approach. From my viewpoint, this way is preferred for Senior Players. :)
Nguyen Huu Khoi
On Tue, Dec 19, 2023 at 1:32 AM Oliver Weinmann oliver.weinmann@me.com wrote:
Hi Satish,
If you are using kolla-Ansible, you can have a look at my blog:
https://www.roksblog.de/openstack-magnum-cluster-api-driver/
Im not sure if the steps work yet. It was very late yesterday when I wrote the article and at first it looked as if it was working but I failed to deploy the cluster.
Cheers, Oliver
Von meinem iPhone gesendet
Am 18.12.2023 um 17:25 schrieb Satish Patel satish.txt@gmail.com:
Little progress, now I am stuck here.
Error: admission webhook " validation.clusterresourceset.addons.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the
# openstack coe cluster show j5
+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ | Field | Value
|+----------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------+ | status | CREATE_FAILED
|| health_status | None
|| cluster_template_id | 2ddfdf2a-5962-42ee-a2e2-ff9e0b19dbe5
|| node_addresses | []
|| uuid | 96723c28-65d2-4add-a29a-cf47a78e32a2
|| stack_id | kube-a8dvb
|| status_reason | admission webhook " validation.clusterresourceset.addons.cluster.x-k8s.io" denied the request: spec: Forbidden: can be set only if the |
On Mon, Dec 18, 2023 at 9:53 AM Oliver Weinmann oliver.weinmann@me.com wrote:
Im focusing purely on kolla-ansible. For myself it would be very helpful to have all info in a single place, so I assume other would also benefit from it.
Von meinem iPhone gesendet
Am 18.12.2023 um 11:11 schrieb Nguyễn Hữu Khôi <nguyenhuukhoinw@gmail.com
:
This is not about doc much. Because we have too many cloud environments. If we have a good blog, are you sure what you write will work well with all environments? Nguyen Huu Khoi
On Mon, Dec 18, 2023 at 5:01 PM Satish Patel satish.txt@gmail.com wrote:
Hi Oliver,
I am 100% with you, There isn't any good technical blog or document about integration with magnum-capi drivers. There is good information but not good technical details with simple steps to follow. How to troubleshoot components etc. for the last few days I am struggling and still have no luck. I have tried all possible combinations. If my setup works then I will surely write good blogs.
On Mon, Dec 18, 2023 at 3:33 AM Oliver Weinmann oliver.weinmann@me.com wrote:
Hi all,
I’m also trying to get the Vexxhost CAPI driver working under Kolla-Ansible. Many thanks to Nguyen Huu Khoi Github page. This was a very good starting point. My goal is to collect all the info to get it working in a single place (my blog). Currently the info is pretty much scattered on different websites, I managed to create the cluster template but the creation fails immediately. This is the error that I get in magnum-conductor.log:
==> /var/log/kolla/magnum/magnum-conductor.log <== 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/magnum/service/periodic.py", line 100, in update_status 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall ng.destroy() 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall return fn(self, *args, **kwargs) 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/magnum/objects/nodegroup.py", line 175, in destroy 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall self.dbapi.destroy_nodegroup(self.cluster_id, self.uuid) 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall File "/var/lib/kolla/venv/lib64/python3.9/site-packages/magnum/db/sqlalchemy/api.py", line 832, in destroy_nodegroup 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall raise exception.NodeGroupNotFound(nodegroup=nodegroup_id) 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall magnum.common.exception.NodeGroupNotFound: Nodegroup 4277e9e6-5c3e-4cce-a1cf-1f5e8c2f0689 could not be found. 2023-12-17 23:27:03.482 7 ERROR oslo.service.loopingcall
(2023.1) [vagrant@seed ~]$ openstack coe cluster list
+--------------------------------------+--------------------------------------+---------+------------+--------------+-----------------+---------------+ | uuid | name | keypair | node_count | master_count | status | health_status |
+--------------------------------------+--------------------------------------+---------+------------+--------------+-----------------+---------------+ | b4ce540f-78a9-4c5d-a687-e992b3bd19a7 | k8s-flan-small-37-v1.23.3-containerd | mykey | 2 | 1 | CREATE_COMPLETE | HEALTHY | | e8acc6da-f937-4e8f-9df8-1728a8079ed0 | k8s-v1.24.16 | mykey | 2 | 1 | CREATE_FAILED | None |
+--------------------------------------+--------------------------------------+---------+------------+--------------+-----------------+---------------+ (2023.1) [vagrant@seed ~]$ openstack coe nodegroup list k8s-v1.24.16
+--------------------------------------+----------------+---------------------+--------------------------------------+------------+--------------------+--------+ | uuid | name | flavor_id | image_id | node_count | status | role |
+--------------------------------------+----------------+---------------------+--------------------------------------+------------+--------------------+--------+ | 21c10537-e3d3-44cf-8e58-731cfeb5b9fe | default-master | m1.kubernetes.small | 9d989f56-359b-4d6a-a914-926e0ea938d7 | 1 | CREATE_IN_PROGRESS | master | | 12dec017-38cc-42d9-b944-649ae356907d | default-worker | m1.kubernetes.small | 9d989f56-359b-4d6a-a914-926e0ea938d7 | 2 | CREATE_IN_PROGRESS | worker |
+--------------------------------------+----------------+---------------------+--------------------------------------+------------+--------------------+————+
Cheers, Oliver
[On 17. Dec 2023, at 22:43, kmceliker@gmail.com wrote:
Here is an example of a CAPI deployment code for OpenStack, mate - using the clusterctl tool and the cluster-template.yaml file. This code will create a cluster named capi-openstack with one control plane node and three worker nodes, using the ubuntu-2204 image and the m1.medium flavor. You need to replace the placeholders with your own values
# Install clusterctl curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.0.1/clus... -o clusterctl chmod +x ./clusterctl sudo mv ./clusterctl /usr/local/bin/clusterctl
# Set environment variables export OPENSTACK_CLOUD=<openstack-cloud> export OPENSTACK_USERNAME=<openstack-username> export OPENSTACK_PASSWORD=<openstack-password> export OPENSTACK_DOMAIN_NAME=<openstack-domain-name> export OPENSTACK_PROJECT_ID=<openstack-project-id> export OPENSTACK_SSH_KEY_NAME=<openstack-ssh-key-name> export OPENSTACK_DNS_NAMESERVERS=<openstack-dns-nameservers> export OPENSTACK_EXTERNAL_NETWORK_ID=<openstack-external-network-id> export OPENSTACK_CONTROL_PLANE_MACHINE_FLAVOR=m1.medium export OPENSTACK_NODE_MACHINE_FLAVOR=m1.medium export OPENSTACK_IMAGE_NAME=ubuntu-2204 export KUBERNETES_VERSION=v1.23.15
# Initialize clusterctl clusterctl init --infrastructure openstack
# Create cluster clusterctl config cluster capi-openstack --kubernetes-version $KUBERNETES_VERSION --control-plane-machine-count=1 --worker-machine-count=3 > cluster-template.yaml clusterctl create cluster --kubeconfig ~/.kube/config --infrastructure openstack:v0.6.0 --bootstrap kubeadm:v0.4.4 --control-plane kubeadm:v0.4.4 --cluster capi-openstack --namespace default --from cluster-template.yaml
Also you can turn on Enable to Magnum Report Log on OpenStack to provide with us or Take a look deep dively as the following link; https://github.com/kubernetes-sigs/cluster-api-provider-openstack
Best, Kerem Çeliker Head of Cloud Architecture tr.linkedin.com/in/keremceliker
On 2023-12-19 04:31:33 +0700 (+0700), Nguyễn Hữu Khôi wrote: [...]
I think Openstack still need an OFFICIAL Forum.I talked about this some months ago. New users cannot search problems via email, a very hard approach.
[...]
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... does function as a real web forum now. It has list-specific and site-wide keyword search indexing updated in real time, and you can read and post to it from a browser without using any E-mail client if you prefer that kind of interaction. It even has upvote/downvote buttons for "liking" posts, read/unread tracking, favorites, profiles... I'm sure it's not perfect, but it's where most of the knowledgeable OpenStack people are reading and responding.
We could create a separate web-only forum where people aren't allowed to treat it as a mailing list, and then anyone asking questions there will get answers just from people who are paying attention to that forum. We tried it in the past (three different times in the past 13 years), but each time we ended up with a useless forum full of questions nobody was answering. It was the easiest place for newcomers to ask questions, however people with experience in the software had no reason to actually pay attention and answer anything in it. The end result was a much *worse* and far more frustrating user experience, because a forum can have all the fancy bells and whistles but if you never get any (or any useful) answers then what's the point?
Oh. Why I didn't see it before, some months ago, is very basic. It has changed, I am happy with this now.
But It will be much better if we: - separate topic: ask and answer, tutorial... - improve mail size limit. When I reply with many previous loops. I need approment from mods - allow attach pictures (size limit).
I get what you are saying. Thank you for your explanation.
Nguyen Huu Khoi
On Tue, Dec 19, 2023 at 5:57 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2023-12-19 04:31:33 +0700 (+0700), Nguyễn Hữu Khôi wrote: [...]
I think Openstack still need an OFFICIAL Forum.I talked about this some months ago. New users cannot search problems via email, a very hard approach.
[...]
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... does function as a real web forum now. It has list-specific and site-wide keyword search indexing updated in real time, and you can read and post to it from a browser without using any E-mail client if you prefer that kind of interaction. It even has upvote/downvote buttons for "liking" posts, read/unread tracking, favorites, profiles... I'm sure it's not perfect, but it's where most of the knowledgeable OpenStack people are reading and responding.
We could create a separate web-only forum where people aren't allowed to treat it as a mailing list, and then anyone asking questions there will get answers just from people who are paying attention to that forum. We tried it in the past (three different times in the past 13 years), but each time we ended up with a useless forum full of questions nobody was answering. It was the easiest place for newcomers to ask questions, however people with experience in the software had no reason to actually pay attention and answer anything in it. The end result was a much *worse* and far more frustrating user experience, because a forum can have all the fancy bells and whistles but if you never get any (or any useful) answers then what's the point? -- Jeremy Stanley
Folks,
I have made very good progress, now I can see CAPI started building workload cluster but when I noticed its stuck in CREATE_IN_PROGRESS for long time then I hope into workload master node and noticed following error spitting up in logs
Look like some cinder-csi related error (I am using 1.27.x image to build cluster)
38u-xpvv4-fkp9f kubelet[1815]: I1219 21:30:22.631478 1815 scope.go:115] "RemoveContainer" containerID="00c2df6cf4e1e4cac15d0d2698fbe19adedb7c5b49c7102c99e3b273ef57335b" Dec 19 21:30:22 kube-8f38u-xpvv4-fkp9f kubelet[1815]: E1219 21:30:22.632141 1815 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to "StartContainer" for "openstack-cloud-controller-manager" with CrashLoopBackOff: "back-off 5m0s restarting failed container=openstack-cloud-controller-manager pod=openstack-cloud-controller-manager-fhnhj_kube-system(7c1162c6-51d0-4207-b462-97e2536d302d)"" pod="kube-system/openstack-cloud-controller-manager-fhnhj" podUID=7c1162c6-51d0-4207-b462-97e2536d302d Dec 19 21:30:26 kube-8f38u-xpvv4-fkp9f kubelet[1815]: I1219 21:30:26.631784 1815 scope.go:115] "RemoveContainer" containerID="7f60974870b0eb472549de51777ea6660db2592687eafa05f2dcbd6db25bdfb1" Dec 19 21:30:26 kube-8f38u-xpvv4-fkp9f kubelet[1815]: E1219 21:30:26.632424 1815 pod_workers.go:1294] "Error syncing pod, skipping" err="failed to "StartContainer" for "cinder-csi-plugin" with CrashLoopBackOff: "back-off 5m0s restarting failed container=cinder-csi-plugin pod=csi-cinder-nodeplugin-9fjwp_kube-system(0b8f67de-fdf2-4996-aab4-5998da795434)"" pod="kube-system/csi-cinder-nodeplugin-9fjwp" podUID=0b8f67de-fdf2-4996-aab4-5998da795434
root@kube-8f38u-xpvv4-fkp9f:/etc/kubernetes# kubectl get deployments -A NAMESPACE NAME READY UP-TO-DATE AVAILABLE AGE kube-system calico-kube-controllers 0/1 1 0 23m kube-system coredns 0/2 2 0 23m kube-system csi-cinder-controllerplugin 0/1 1 0 23m
root@kube-8f38u-xpvv4-fkp9f:/etc/kubernetes# kubectl get nodes NAME STATUS ROLES AGE VERSION kube-8f38u-default-worker-zpg2j-9w5pk Ready <none> 22m v1.27.4 kube-8f38u-default-worker-zpg2j-vp56f Ready <none> 22m v1.27.4 kube-8f38u-xpvv4-fkp9f Ready control-plane 24m v1.27.4
What else should I check or look for?
On Mon, Dec 18, 2023 at 7:39 PM Nguyễn Hữu Khôi nguyenhuukhoinw@gmail.com wrote:
Oh. Why I didn't see it before, some months ago, is very basic. It has changed, I am happy with this now.
But It will be much better if we:
- separate topic: ask and answer, tutorial...
- improve mail size limit. When I reply with many previous loops. I need
approment from mods
- allow attach pictures (size limit).
I get what you are saying. Thank you for your explanation.
Nguyen Huu Khoi
On Tue, Dec 19, 2023 at 5:57 AM Jeremy Stanley fungi@yuggoth.org wrote:
On 2023-12-19 04:31:33 +0700 (+0700), Nguyễn Hữu Khôi wrote: [...]
I think Openstack still need an OFFICIAL Forum.I talked about this some months ago. New users cannot search problems via email, a very hard approach.
[...]
https://lists.openstack.org/archives/list/openstack-discuss@lists.openstack.... does function as a real web forum now. It has list-specific and site-wide keyword search indexing updated in real time, and you can read and post to it from a browser without using any E-mail client if you prefer that kind of interaction. It even has upvote/downvote buttons for "liking" posts, read/unread tracking, favorites, profiles... I'm sure it's not perfect, but it's where most of the knowledgeable OpenStack people are reading and responding.
We could create a separate web-only forum where people aren't allowed to treat it as a mailing list, and then anyone asking questions there will get answers just from people who are paying attention to that forum. We tried it in the past (three different times in the past 13 years), but each time we ended up with a useless forum full of questions nobody was answering. It was the easiest place for newcomers to ask questions, however people with experience in the software had no reason to actually pay attention and answer anything in it. The end result was a much *worse* and far more frustrating user experience, because a forum can have all the fancy bells and whistles but if you never get any (or any useful) answers then what's the point? -- Jeremy Stanley
participants (4)
-
Jeremy Stanley
-
Nguyễn Hữu Khôi
-
Oliver Weinmann
-
Satish Patel