Open Stack

We are ecstatic to announce the release of:

magnum 10.1.0: Container Management project for OpenStack

This release is part of the ussuri stable release series.

The source is available from:

    https://opendev.org/openstack/magnum

Download the package from:

    https://tarballs.openstack.org/magnum/

Please report issues through:

    https://bugs.launchpad.net/magnum/+bugs

For more details, please see below.

10.1.0
^^^^^^


New Features
************

* Users can enable or disable master_lb_enabled when creating a
  cluster.

* The default 10 seconds health polling interval is too frequent for
  most of the cases. Now it has been changed to 60s. A new config
  *health_polling_interval* is supported to make the interval
  configurable. Cloud admin can totally disable the health polling by
  set a negative value for the config.

* Expose autoscaler prometheus metrics on pod port metrics (8085).

* Add a new label named *master_lb_allowed_cidrs* to control the IP
  ranges which can access the k8s API and etcd load balancers of
  master. To get this feature, the minimum version of Heat is
  stable/ussuri and minimum version of Octavia is stable/train.

* A new boolean flag is introduced in the CLuster and Nodegroup
  create API calls. Using this flag, users can override label values
  when clusters or nodegroups are created without having to specify
  all the inherited values. To do that, users have to specify the
  labels with their new values and use the flag --merge-labels. At the
  same time, three new fields are added in the cluster and nodegroup
  show outputs, showing the differences between the actual and the
  iherited labels.

* Magnum now cascade deletes all the load balancers before deleting
  the cluster, not only including load balancers for the cluster
  services and ingresses, but also those for Kubernetes API/etcd
  endpoints.

* Support Helm v3 client to install helm charts. To use this
  feature, users will need to use helm_client_tag>=v3.0.0 (default
  helm_client_tag=v3.2.1). All the existing chart used to depend on
  Helm v2, e.g. nginx ingress controller, metrics server, prometheus
  operator and prometheus adapter are now also installable using v3
  client. Also introduce helm_client_sha256 and helm_client_url that
  users can specify to install non-default helm client version
  (https://github.com/helm/helm/releases).

* Cloud admin user now can do rolling upgrade on behalf of end user
  so as to do urgent security patching when it's necessary.

* Add to prometheus federation exported metrics the cluster_uuid
  label.


Upgrade Notes
*************

* If it's still preferred to have 10s health polling interval for
  Kubernetes cluster. It can be set by config
  *health_polling_interval* under *kubernetes* section.

* Now the default admission controller list is updated by as
  "NodeRestriction, PodSecurityPolicy, NamespaceLifecycle,
  LimitRanger, ServiceAccount, ResourceQuota, TaintNodesByCondition,
  Priority, DefaultTolerationSeconds, DefaultStorageClass,
  StorageObjectInUseProtection, PersistentVolumeClaimResize,
  MutatingAdmissionWebhook, ValidatingAdmissionWebhook, RuntimeClass"

* Default tiller_tag is set to v2.16.7. The charts remain compatible
  but helm_client_tag will also need to be set to the same value as
  tiller_tag, i.e. v2.16.7. In this case, the user will also need to
  provide helm_client_sha256 for the helm client binary intended for
  use.

* Bumped prometheus-operator chart tag to 8.12.13. Added
  container_infra_prefix to missing prometheusOperator images.


Deprecation Notes
*****************

* Support for Helm v2 client will be removed in X release.


Bug Fixes
*********

* Deploy traefik from the heat-agent

  Use kubectl from the heat agent to apply the traefik deployment.
  Current behaviour was to create a systemd unit to send the manifests
  to the API.

  This way we will have only one way for applying manifests to the
  API.

  This change is triggered to adddress the kubectl change [0] that is
  not using 127.0.0.1:8080 as the default kubernetes API.

  [0] https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/
  CHANGELOG-1.18.md#kubectl

* Fixes an edge case where when a cluster with additional nodegroups
  is patched with health_status and health_status_reason, it was
  leading to the default-worker nodegroup being resized.

* Fixes a regression which left behind trustee user accounts and
  certificates when a cluster is deleted.

* Now the label *fixed_network_cidr* have been renamed with
  *fixed_subnet_cidr*. And it can be passed in and set correctly.

* Fix an issue with private clusters getting stuck in
  CREATE_IN_PROGRESS status where floating_ip_enabled=True in the
  cluster template but this is disabled when the cluster is created.

* Fixes database migrations with SQLAlchemy 1.3.20.

* Prometheus server now scrape metrics from traefik proxy.
  Prometheus server now scrape metrics from cluster autoscaler.

* Scrape metrics from kube-{controller-manager,scheduler}. Disable
  PrometheusRule for etcd.

* Fixes an issue with cluster deletion if load balancers do not
  exist. See *story 2008548
  <https://storyboard.openstack.org/#!/story/2008548>* for details.

Changes in magnum 10.0.0..10.1.0
--------------------------------

ffbdbbc0 Revert "Update containerd version and tarball URL"
a4923254 Fix debug logging during cluster upgrade
850b94aa Do not create constraints for boolean fields
2f217810 Lower log level of missing output
8301a180 Re-use transport for rpc server
1055bf2d Deploy traefik from the heat-agent
5c47db10 Remove shebang from scripts
2b353f58 [k8s] Fix default admission controller
2473f226 Re-use transport for rpc calls
92f518df k8s-fcos: Source bashrc for clusterconfig
92260bf0 [k8s-fcos] Fix insecure registry
caedd838 k8s: Do not use insecure api port
9a21fe75 Fix cluster deletion when load balancers don't exist
d3f7445e Make kubelet and kube-proxy use the secure port
344f5059 Fix validation for master_lb_enabled
6696b057 Update containerd version and tarball URL
09d1fefb Update helm charts origin repository
4393147b Add image prefix for grafana images
63ab64ba Use kube_master_ip for monitoring when no floating ip is used
33e96f50 Fix Cinder CSI
15b5f970 Fix misquoted comment
93a47e76 Fix database migrations
0c95ef1a Update default k8s admission controller list
54b36190 Drop KUBE_API_PORT for kube-apiserver
5ce4ca05 Remove cloud-config from k8s worker node
8744130e Fix syntax error in default rolesync configmap
74b67698 Stop using delete_on_termination for BFV instances
06635a3f Remove duplicated etcd_volume_size param in coreos template
a9bdf93d Configure placeholder role-mapping Sync
9faa4b3d Remove warning for scale_manager
2e6ebc0a Drop lower constraints testing
fe35af8e Drop dockerhub password from stable/ussuri
5bd16d9d [fix] Sync nodegroup status before delete_complete
c0fabb92 Update default values for docker nofile and vm.max_map_count
3e4cac2b [k8s-atomic] Support master_lb_allowed_cidrs in template
34468cf0 [fix] Append v3/v1 to auth_url/magnum_url if discovery fails
e68f1d85 Fix proxy issue for etcd and k8s
53f6de60 Add master_lb_enabled to cluster
b699e0c9 [k8s] Use helm upgrade --install in deployment loop
bb580533 Fix ServerAddressOutputMapping for private clusters
1e99f41d [k8s] Fix PreDeletionFailed if Heat stack is missing
3b428881 Fix label fixed_network_cidr
e0fecc1d [ci] Fix gate by installing python3-docker
bcffb630 [K8S] Delete all related load balancers before deleting cluster
7794e3f8 resize: Send only nodes_to_remove and node_count
57aab5a0 [k8s] Add label 'master_lb_allowed_cidrs'
709c448f [fix] Use default_ng_worker.node_count for patches
09acf980 More verbose logs for cluster ops
9b8859be [hca] Use ussuri-stable-1 as default for stable/ussuri
6f6e3a83 Use full name for hyperkube image inspect
b90dff5b Support proxy for helm install
a3942670 Support upgrade on behalf of user by admin
49e2468e Fix small issues rolling upgrade
be786650 api: Do not guess based on name extension
ca058f89 [k8s] Use Helm v3 by default
5ae48c26 Scrape internal kubernetes components
08202e80 Update prometheus monitoring chart and images
6191c93e Scrape traefik and autoscaler metrics
1309a829 [k8s] Expose autoscaler prometheus metrics
13944e9c k8s: Add admin.conf kubeconfig
a94d672e k8s: Use the same kubectl version as API
4430329c atomic: Do not install control-plane on minions
b782325d [k8s] Update Cluster Autoscaler ClusterRole
56ed41ea Monkey patch original current_thread _active
f8a89b1b Add newline to fix E004 bashate error
dbe461c1 [k8s] Support configurable health polling interval
b5ec9545 Labels override
dc33089f Update nginx-ingress to v1.36.3 and 0.32.0 tag


Diffstat (except docs and test files)
-------------------------------------

.zuul.yaml                                         |  27 --
.../dcos_centos_v1/templates/dcoscluster.yaml      |   4 +-
contrib/drivers/heat/dcos_centos_template_def.py   |   8 +-
contrib/drivers/k8s_opensuse_v1/template_def.py    |   6 +-
.../k8s_opensuse_v1/templates/kubecluster.yaml     |   4 +-
magnum/api/app.py                                  |   1 +
magnum/api/attr_validator.py                       |   2 +-
magnum/api/controllers/v1/bay.py                   |   5 +
magnum/api/controllers/v1/cluster.py               |  61 +++-
magnum/api/controllers/v1/cluster_actions.py       |   5 +
magnum/api/controllers/v1/nodegroup.py             |  33 ++
magnum/api/utils.py                                |  21 ++
magnum/cmd/__init__.py                             |   6 +
magnum/common/neutron.py                           |   4 +-
magnum/common/octavia.py                           | 103 ++++--
magnum/common/policies/cluster.py                  |  12 +
magnum/common/rpc.py                               |  10 -
magnum/common/rpc_service.py                       |  22 +-
magnum/conductor/api.py                            |   5 +-
magnum/conductor/scale_manager.py                  |   8 +-
magnum/conf/kubernetes.py                          |   5 +
.../1d045384b966_add_insecure_baymodel_attr.py     |   9 +-
...304554e2_adding_magnum_service_functionality.py |   8 +-
...3c9c6191_add_public_column_to_baymodel_table.py |   4 +-
...98132c7_change_cluster_to_support_nodegroups.py |   2 +-
...380964133d_add_network_subnet_fip_to_cluster.py |   2 +-
.../versions/4e263f236334_add_registry_enabled.py  |   2 +-
.../versions/5ad410481b88_rename_insecure.py       |   2 +-
...d_master_lb_enabled_column_to_baymodel_table.py |   4 +-
.../87e62e3c7abc_add_hidden_to_cluster_template.py |   6 +-
...5096e2334ee_add_master_lb_enabled_to_cluster.py |  43 +++
.../versions/ac92cbae311c_add_nodegoup_table.py    |   4 +-
...612248cab_add_floating_ip_enabled_column_to_.py |   2 +-
magnum/db/sqlalchemy/models.py                     |   1 +
magnum/drivers/common/k8s_monitor.py               |   2 +-
.../fragments/configure-docker-registry.sh         |   2 +-
...onfigure_docker_storage_driver_fedora_coreos.sh |   3 +
.../templates/kubernetes/fragments/add-proxy.sh    |   2 -
.../kubernetes/fragments/calico-service-v3-3-x.sh  |   4 +-
.../kubernetes/fragments/calico-service.sh         |   7 +-
.../kubernetes/fragments/configure-etcd.sh         |   3 +-
.../fragments/configure-kubernetes-master.sh       | 106 +++---
.../fragments/configure-kubernetes-minion.sh       |  17 +-
.../kubernetes/fragments/core-dns-service.sh       |   4 +-
.../kubernetes/fragments/disable-selinux.sh        |   1 -
.../kubernetes/fragments/enable-auto-healing.sh    |   4 +-
.../kubernetes/fragments/enable-auto-scaling.sh    |  72 ++--
.../fragments/enable-cert-api-manager.sh           |   2 -
.../kubernetes/fragments/enable-cinder-csi.sh      |  13 +-
.../kubernetes/fragments/enable-helm-tiller.sh     |   4 +-
.../fragments/enable-ingress-controller.sh         |   2 -
.../kubernetes/fragments/enable-ingress-octavia.sh |   2 +-
.../kubernetes/fragments/enable-ingress-traefik.sh |  48 +--
.../kubernetes/fragments/enable-keystone-auth.sh   |  17 +-
.../fragments/enable-prometheus-monitoring.sh      |   6 +-
.../kubernetes/fragments/enable-services-master.sh |   6 +-
.../kubernetes/fragments/enable-services-minion.sh |   2 -
.../kubernetes/fragments/flannel-service.sh        |   4 +-
.../kubernetes/fragments/install-clients.sh        |  37 ++
.../templates/kubernetes/fragments/install-cri.sh  |   4 +-
.../kubernetes/fragments/install-helm-modules.sh   |  97 +++++-
.../fragments/kube-apiserver-to-kubelet-role.sh    |   4 +-
.../kubernetes/fragments/kube-dashboard-service.sh |   4 +-
.../kubernetes/fragments/make-cert-client.sh       |   2 -
.../templates/kubernetes/fragments/make-cert.sh    |  20 +-
.../kubernetes/fragments/start-container-agent.sh  |   2 -
.../kubernetes/fragments/upgrade-kubernetes.sh     |  49 ++-
.../kubernetes/fragments/wc-notify-master.sh       |   4 +-
.../fragments/write-heat-params-master.sh          |   4 +-
.../kubernetes/fragments/write-heat-params.sh      |   2 -
.../kubernetes/fragments/write-kube-os-config.sh   |   3 -
.../templates/kubernetes/helm/ingress-nginx.sh     | 369 ++++++++------------
.../templates/kubernetes/helm/metrics-server.sh    | 104 +-----
.../kubernetes/helm/prometheus-adapter.sh          | 141 ++------
.../kubernetes/helm/prometheus-operator.sh         | 384 ++++++++++++---------
magnum/drivers/common/templates/lb_api.yaml        |  28 +-
magnum/drivers/common/templates/lb_etcd.yaml       |  28 +-
magnum/drivers/heat/driver.py                      |  83 +++--
magnum/drivers/heat/k8s_coreos_template_def.py     |   4 +-
magnum/drivers/heat/k8s_fedora_template_def.py     |   5 +-
magnum/drivers/heat/k8s_template_def.py            |  52 ++-
magnum/drivers/heat/swarm_fedora_template_def.py   |  10 +-
magnum/drivers/heat/swarm_mode_template_def.py     |  27 +-
magnum/drivers/heat/template_def.py                |  38 +-
.../k8s_coreos_v1/templates/kubecluster.yaml       |  10 +-
.../templates/kubecluster.yaml                     |  48 ++-
.../k8s_fedora_atomic_v1/templates/kubemaster.yaml |  32 +-
.../k8s_fedora_atomic_v1/templates/kubeminion.yaml |   9 +-
.../templates/fcct-config.yaml                     |  22 +-
.../templates/kubecluster.yaml                     |  53 ++-
.../k8s_fedora_coreos_v1/templates/kubemaster.yaml |  40 ++-
.../k8s_fedora_coreos_v1/templates/kubeminion.yaml |  18 +-
.../k8s_fedora_coreos_v1/templates/user_data.json  |  45 ++-
magnum/drivers/mesos_ubuntu_v1/template_def.py     |   7 +-
.../mesos_ubuntu_v1/templates/mesoscluster.yaml    |   4 +-
.../swarm_fedora_atomic_v1/templates/cluster.yaml  |   6 +-
.../templates/swarmcluster.yaml                    |   6 +-
magnum/objects/cluster.py                          |   4 +-
magnum/service/periodic.py                         |   4 +-
.../api/controllers/v1/test_cluster_actions.py     |  39 ++-
.../unit/api/controllers/v1/test_nodegroup.py      |  78 +++++
.../handlers/test_k8s_cluster_conductor.py         |  81 ++++-
.../handlers/test_mesos_cluster_conductor.py       |   4 +
.../handlers/test_swarm_cluster_conductor.py       |   4 +
playbooks/container-builder-setup-gate.yaml        |   6 +-
...aefik-from-the-heat-agent-0bb32f0f2c97405d.yaml |  18 +
...ter_lb_enabled-to-cluster-c773fac9086b2531.yaml |   5 +
...s-health-polling-interval-75bb83b4701d48c5.yaml |  13 +
...ault-admission-controller-04398548cf63597c.yaml |   5 +
...ault-ng-worker-node-count-a88911a0b7a760a7.yaml |   6 +
.../ensure-delete-complete-2f9bb53616e1e02b.yaml   |   5 +
...expose_autoscaler_metrics-0ea9c61660409efe.yaml |   4 +
...-label-fixed_network_cidr-95d6a2571b58a8fc.yaml |   6 +
...ping-for-private-clusters-73a874bb4827d568.yaml |   6 +
.../master-lb-allowed-cidrs-cc599da4eb96e983.yaml  |   7 +
.../notes/merge-labels-9ba7deffc5bb3c7f.yaml       |  10 +
.../notes/migrations-1.3.20-60e5f990422f2ca5.yaml  |   4 +
...ing_scrape_ca_and_traefik-5544d8dd5ab7c234.yaml |   5 +
...onitoring_scrape_internal-6697e50f091b0c9c.yaml |   5 +
...-delete-all-loadbalancers-350a69ec787e11ea.yaml |   5 +
.../notes/story-2008548-65a571ad15451937.yaml      |   6 +
.../notes/support-helm-v3-5c68eca89fc9446b.yaml    |  19 +
...upgrade-on-behalf-of-user-c04994831360f8c1.yaml |   5 +
...ate_prometheus_monitoring-342a86f826be6579.yaml |   8 +
135 files changed, 2240 insertions(+), 1385 deletions(-)