We are ecstatic to announce the release of: magnum 10.1.0: Container Management project for OpenStack This release is part of the ussuri stable release series. The source is available from: https://opendev.org/openstack/magnum Download the package from: https://tarballs.openstack.org/magnum/ Please report issues through: https://bugs.launchpad.net/magnum/+bugs For more details, please see below. 10.1.0 ^^^^^^ New Features ************ * Users can enable or disable master_lb_enabled when creating a cluster. * The default 10 seconds health polling interval is too frequent for most of the cases. Now it has been changed to 60s. A new config *health_polling_interval* is supported to make the interval configurable. Cloud admin can totally disable the health polling by set a negative value for the config. * Expose autoscaler prometheus metrics on pod port metrics (8085). * Add a new label named *master_lb_allowed_cidrs* to control the IP ranges which can access the k8s API and etcd load balancers of master. To get this feature, the minimum version of Heat is stable/ussuri and minimum version of Octavia is stable/train. * A new boolean flag is introduced in the CLuster and Nodegroup create API calls. Using this flag, users can override label values when clusters or nodegroups are created without having to specify all the inherited values. To do that, users have to specify the labels with their new values and use the flag --merge-labels. At the same time, three new fields are added in the cluster and nodegroup show outputs, showing the differences between the actual and the iherited labels. * Magnum now cascade deletes all the load balancers before deleting the cluster, not only including load balancers for the cluster services and ingresses, but also those for Kubernetes API/etcd endpoints. * Support Helm v3 client to install helm charts. To use this feature, users will need to use helm_client_tag>=v3.0.0 (default helm_client_tag=v3.2.1). All the existing chart used to depend on Helm v2, e.g. nginx ingress controller, metrics server, prometheus operator and prometheus adapter are now also installable using v3 client. Also introduce helm_client_sha256 and helm_client_url that users can specify to install non-default helm client version (https://github.com/helm/helm/releases). * Cloud admin user now can do rolling upgrade on behalf of end user so as to do urgent security patching when it's necessary. * Add to prometheus federation exported metrics the cluster_uuid label. Upgrade Notes ************* * If it's still preferred to have 10s health polling interval for Kubernetes cluster. It can be set by config *health_polling_interval* under *kubernetes* section. * Now the default admission controller list is updated by as "NodeRestriction, PodSecurityPolicy, NamespaceLifecycle, LimitRanger, ServiceAccount, ResourceQuota, TaintNodesByCondition, Priority, DefaultTolerationSeconds, DefaultStorageClass, StorageObjectInUseProtection, PersistentVolumeClaimResize, MutatingAdmissionWebhook, ValidatingAdmissionWebhook, RuntimeClass" * Default tiller_tag is set to v2.16.7. The charts remain compatible but helm_client_tag will also need to be set to the same value as tiller_tag, i.e. v2.16.7. In this case, the user will also need to provide helm_client_sha256 for the helm client binary intended for use. * Bumped prometheus-operator chart tag to 8.12.13. Added container_infra_prefix to missing prometheusOperator images. Deprecation Notes ***************** * Support for Helm v2 client will be removed in X release. Bug Fixes ********* * Deploy traefik from the heat-agent Use kubectl from the heat agent to apply the traefik deployment. Current behaviour was to create a systemd unit to send the manifests to the API. This way we will have only one way for applying manifests to the API. This change is triggered to adddress the kubectl change [0] that is not using 127.0.0.1:8080 as the default kubernetes API. [0] https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/ CHANGELOG-1.18.md#kubectl * Fixes an edge case where when a cluster with additional nodegroups is patched with health_status and health_status_reason, it was leading to the default-worker nodegroup being resized. * Fixes a regression which left behind trustee user accounts and certificates when a cluster is deleted. * Now the label *fixed_network_cidr* have been renamed with *fixed_subnet_cidr*. And it can be passed in and set correctly. * Fix an issue with private clusters getting stuck in CREATE_IN_PROGRESS status where floating_ip_enabled=True in the cluster template but this is disabled when the cluster is created. * Fixes database migrations with SQLAlchemy 1.3.20. * Prometheus server now scrape metrics from traefik proxy. Prometheus server now scrape metrics from cluster autoscaler. * Scrape metrics from kube-{controller-manager,scheduler}. Disable PrometheusRule for etcd. * Fixes an issue with cluster deletion if load balancers do not exist. See *story 2008548 <https://storyboard.openstack.org/#!/story/2008548>* for details. Changes in magnum 10.0.0..10.1.0 -------------------------------- ffbdbbc0 Revert "Update containerd version and tarball URL" a4923254 Fix debug logging during cluster upgrade 850b94aa Do not create constraints for boolean fields 2f217810 Lower log level of missing output 8301a180 Re-use transport for rpc server 1055bf2d Deploy traefik from the heat-agent 5c47db10 Remove shebang from scripts 2b353f58 [k8s] Fix default admission controller 2473f226 Re-use transport for rpc calls 92f518df k8s-fcos: Source bashrc for clusterconfig 92260bf0 [k8s-fcos] Fix insecure registry caedd838 k8s: Do not use insecure api port 9a21fe75 Fix cluster deletion when load balancers don't exist d3f7445e Make kubelet and kube-proxy use the secure port 344f5059 Fix validation for master_lb_enabled 6696b057 Update containerd version and tarball URL 09d1fefb Update helm charts origin repository 4393147b Add image prefix for grafana images 63ab64ba Use kube_master_ip for monitoring when no floating ip is used 33e96f50 Fix Cinder CSI 15b5f970 Fix misquoted comment 93a47e76 Fix database migrations 0c95ef1a Update default k8s admission controller list 54b36190 Drop KUBE_API_PORT for kube-apiserver 5ce4ca05 Remove cloud-config from k8s worker node 8744130e Fix syntax error in default rolesync configmap 74b67698 Stop using delete_on_termination for BFV instances 06635a3f Remove duplicated etcd_volume_size param in coreos template a9bdf93d Configure placeholder role-mapping Sync 9faa4b3d Remove warning for scale_manager 2e6ebc0a Drop lower constraints testing fe35af8e Drop dockerhub password from stable/ussuri 5bd16d9d [fix] Sync nodegroup status before delete_complete c0fabb92 Update default values for docker nofile and vm.max_map_count 3e4cac2b [k8s-atomic] Support master_lb_allowed_cidrs in template 34468cf0 [fix] Append v3/v1 to auth_url/magnum_url if discovery fails e68f1d85 Fix proxy issue for etcd and k8s 53f6de60 Add master_lb_enabled to cluster b699e0c9 [k8s] Use helm upgrade --install in deployment loop bb580533 Fix ServerAddressOutputMapping for private clusters 1e99f41d [k8s] Fix PreDeletionFailed if Heat stack is missing 3b428881 Fix label fixed_network_cidr e0fecc1d [ci] Fix gate by installing python3-docker bcffb630 [K8S] Delete all related load balancers before deleting cluster 7794e3f8 resize: Send only nodes_to_remove and node_count 57aab5a0 [k8s] Add label 'master_lb_allowed_cidrs' 709c448f [fix] Use default_ng_worker.node_count for patches 09acf980 More verbose logs for cluster ops 9b8859be [hca] Use ussuri-stable-1 as default for stable/ussuri 6f6e3a83 Use full name for hyperkube image inspect b90dff5b Support proxy for helm install a3942670 Support upgrade on behalf of user by admin 49e2468e Fix small issues rolling upgrade be786650 api: Do not guess based on name extension ca058f89 [k8s] Use Helm v3 by default 5ae48c26 Scrape internal kubernetes components 08202e80 Update prometheus monitoring chart and images 6191c93e Scrape traefik and autoscaler metrics 1309a829 [k8s] Expose autoscaler prometheus metrics 13944e9c k8s: Add admin.conf kubeconfig a94d672e k8s: Use the same kubectl version as API 4430329c atomic: Do not install control-plane on minions b782325d [k8s] Update Cluster Autoscaler ClusterRole 56ed41ea Monkey patch original current_thread _active f8a89b1b Add newline to fix E004 bashate error dbe461c1 [k8s] Support configurable health polling interval b5ec9545 Labels override dc33089f Update nginx-ingress to v1.36.3 and 0.32.0 tag Diffstat (except docs and test files) ------------------------------------- .zuul.yaml | 27 -- .../dcos_centos_v1/templates/dcoscluster.yaml | 4 +- contrib/drivers/heat/dcos_centos_template_def.py | 8 +- contrib/drivers/k8s_opensuse_v1/template_def.py | 6 +- .../k8s_opensuse_v1/templates/kubecluster.yaml | 4 +- magnum/api/app.py | 1 + magnum/api/attr_validator.py | 2 +- magnum/api/controllers/v1/bay.py | 5 + magnum/api/controllers/v1/cluster.py | 61 +++- magnum/api/controllers/v1/cluster_actions.py | 5 + magnum/api/controllers/v1/nodegroup.py | 33 ++ magnum/api/utils.py | 21 ++ magnum/cmd/__init__.py | 6 + magnum/common/neutron.py | 4 +- magnum/common/octavia.py | 103 ++++-- magnum/common/policies/cluster.py | 12 + magnum/common/rpc.py | 10 - magnum/common/rpc_service.py | 22 +- magnum/conductor/api.py | 5 +- magnum/conductor/scale_manager.py | 8 +- magnum/conf/kubernetes.py | 5 + .../1d045384b966_add_insecure_baymodel_attr.py | 9 +- ...304554e2_adding_magnum_service_functionality.py | 8 +- ...3c9c6191_add_public_column_to_baymodel_table.py | 4 +- ...98132c7_change_cluster_to_support_nodegroups.py | 2 +- ...380964133d_add_network_subnet_fip_to_cluster.py | 2 +- .../versions/4e263f236334_add_registry_enabled.py | 2 +- .../versions/5ad410481b88_rename_insecure.py | 2 +- ...d_master_lb_enabled_column_to_baymodel_table.py | 4 +- .../87e62e3c7abc_add_hidden_to_cluster_template.py | 6 +- ...5096e2334ee_add_master_lb_enabled_to_cluster.py | 43 +++ .../versions/ac92cbae311c_add_nodegoup_table.py | 4 +- ...612248cab_add_floating_ip_enabled_column_to_.py | 2 +- magnum/db/sqlalchemy/models.py | 1 + magnum/drivers/common/k8s_monitor.py | 2 +- .../fragments/configure-docker-registry.sh | 2 +- ...onfigure_docker_storage_driver_fedora_coreos.sh | 3 + .../templates/kubernetes/fragments/add-proxy.sh | 2 - .../kubernetes/fragments/calico-service-v3-3-x.sh | 4 +- .../kubernetes/fragments/calico-service.sh | 7 +- .../kubernetes/fragments/configure-etcd.sh | 3 +- .../fragments/configure-kubernetes-master.sh | 106 +++--- .../fragments/configure-kubernetes-minion.sh | 17 +- .../kubernetes/fragments/core-dns-service.sh | 4 +- .../kubernetes/fragments/disable-selinux.sh | 1 - .../kubernetes/fragments/enable-auto-healing.sh | 4 +- .../kubernetes/fragments/enable-auto-scaling.sh | 72 ++-- .../fragments/enable-cert-api-manager.sh | 2 - .../kubernetes/fragments/enable-cinder-csi.sh | 13 +- .../kubernetes/fragments/enable-helm-tiller.sh | 4 +- .../fragments/enable-ingress-controller.sh | 2 - .../kubernetes/fragments/enable-ingress-octavia.sh | 2 +- .../kubernetes/fragments/enable-ingress-traefik.sh | 48 +-- .../kubernetes/fragments/enable-keystone-auth.sh | 17 +- .../fragments/enable-prometheus-monitoring.sh | 6 +- .../kubernetes/fragments/enable-services-master.sh | 6 +- .../kubernetes/fragments/enable-services-minion.sh | 2 - .../kubernetes/fragments/flannel-service.sh | 4 +- .../kubernetes/fragments/install-clients.sh | 37 ++ .../templates/kubernetes/fragments/install-cri.sh | 4 +- .../kubernetes/fragments/install-helm-modules.sh | 97 +++++- .../fragments/kube-apiserver-to-kubelet-role.sh | 4 +- .../kubernetes/fragments/kube-dashboard-service.sh | 4 +- .../kubernetes/fragments/make-cert-client.sh | 2 - .../templates/kubernetes/fragments/make-cert.sh | 20 +- .../kubernetes/fragments/start-container-agent.sh | 2 - .../kubernetes/fragments/upgrade-kubernetes.sh | 49 ++- .../kubernetes/fragments/wc-notify-master.sh | 4 +- .../fragments/write-heat-params-master.sh | 4 +- .../kubernetes/fragments/write-heat-params.sh | 2 - .../kubernetes/fragments/write-kube-os-config.sh | 3 - .../templates/kubernetes/helm/ingress-nginx.sh | 369 ++++++++------------ .../templates/kubernetes/helm/metrics-server.sh | 104 +----- .../kubernetes/helm/prometheus-adapter.sh | 141 ++------ .../kubernetes/helm/prometheus-operator.sh | 384 ++++++++++++--------- magnum/drivers/common/templates/lb_api.yaml | 28 +- magnum/drivers/common/templates/lb_etcd.yaml | 28 +- magnum/drivers/heat/driver.py | 83 +++-- magnum/drivers/heat/k8s_coreos_template_def.py | 4 +- magnum/drivers/heat/k8s_fedora_template_def.py | 5 +- magnum/drivers/heat/k8s_template_def.py | 52 ++- magnum/drivers/heat/swarm_fedora_template_def.py | 10 +- magnum/drivers/heat/swarm_mode_template_def.py | 27 +- magnum/drivers/heat/template_def.py | 38 +- .../k8s_coreos_v1/templates/kubecluster.yaml | 10 +- .../templates/kubecluster.yaml | 48 ++- .../k8s_fedora_atomic_v1/templates/kubemaster.yaml | 32 +- .../k8s_fedora_atomic_v1/templates/kubeminion.yaml | 9 +- .../templates/fcct-config.yaml | 22 +- .../templates/kubecluster.yaml | 53 ++- .../k8s_fedora_coreos_v1/templates/kubemaster.yaml | 40 ++- .../k8s_fedora_coreos_v1/templates/kubeminion.yaml | 18 +- .../k8s_fedora_coreos_v1/templates/user_data.json | 45 ++- magnum/drivers/mesos_ubuntu_v1/template_def.py | 7 +- .../mesos_ubuntu_v1/templates/mesoscluster.yaml | 4 +- .../swarm_fedora_atomic_v1/templates/cluster.yaml | 6 +- .../templates/swarmcluster.yaml | 6 +- magnum/objects/cluster.py | 4 +- magnum/service/periodic.py | 4 +- .../api/controllers/v1/test_cluster_actions.py | 39 ++- .../unit/api/controllers/v1/test_nodegroup.py | 78 +++++ .../handlers/test_k8s_cluster_conductor.py | 81 ++++- .../handlers/test_mesos_cluster_conductor.py | 4 + .../handlers/test_swarm_cluster_conductor.py | 4 + playbooks/container-builder-setup-gate.yaml | 6 +- ...aefik-from-the-heat-agent-0bb32f0f2c97405d.yaml | 18 + ...ter_lb_enabled-to-cluster-c773fac9086b2531.yaml | 5 + ...s-health-polling-interval-75bb83b4701d48c5.yaml | 13 + ...ault-admission-controller-04398548cf63597c.yaml | 5 + ...ault-ng-worker-node-count-a88911a0b7a760a7.yaml | 6 + .../ensure-delete-complete-2f9bb53616e1e02b.yaml | 5 + ...expose_autoscaler_metrics-0ea9c61660409efe.yaml | 4 + ...-label-fixed_network_cidr-95d6a2571b58a8fc.yaml | 6 + ...ping-for-private-clusters-73a874bb4827d568.yaml | 6 + .../master-lb-allowed-cidrs-cc599da4eb96e983.yaml | 7 + .../notes/merge-labels-9ba7deffc5bb3c7f.yaml | 10 + .../notes/migrations-1.3.20-60e5f990422f2ca5.yaml | 4 + ...ing_scrape_ca_and_traefik-5544d8dd5ab7c234.yaml | 5 + ...onitoring_scrape_internal-6697e50f091b0c9c.yaml | 5 + ...-delete-all-loadbalancers-350a69ec787e11ea.yaml | 5 + .../notes/story-2008548-65a571ad15451937.yaml | 6 + .../notes/support-helm-v3-5c68eca89fc9446b.yaml | 19 + ...upgrade-on-behalf-of-user-c04994831360f8c1.yaml | 5 + ...ate_prometheus_monitoring-342a86f826be6579.yaml | 8 + 135 files changed, 2240 insertions(+), 1385 deletions(-)