<div dir="ltr">You can take a look onto Murano Kubernetes package. There is no autoscaling out of the box, but it will be quite trivial to add a new action for that as there are functions to add new ETC and Kubernetes nodes on master as well as there is a function to add a new VM.<div><br></div><div>Here is an example of a scaleUp action: <a href="https://github.com/gokrokvertskhov/murano-app-incubator/blob/monitoring-ha/io.murano.apps.java.HelloWorldCluster/Classes/HelloWorldCluster.murano#L93">https://github.com/gokrokvertskhov/murano-app-incubator/blob/monitoring-ha/io.murano.apps.java.HelloWorldCluster/Classes/HelloWorldCluster.murano#L93</a></div><div><br></div><div>Here is Kubernetes scaleUp action: <a href="https://github.com/openstack/murano-apps/blob/master/Docker/Kubernetes/KubernetesCluster/package/Classes/KubernetesCluster.yaml#L441">https://github.com/openstack/murano-apps/blob/master/Docker/Kubernetes/KubernetesCluster/package/Classes/KubernetesCluster.yaml#L441</a></div><div><br></div><div>And here is a place where Kubernetes master is update with a new node info: <a href="https://github.com/openstack/murano-apps/blob/master/Docker/Kubernetes/KubernetesCluster/package/Classes/KubernetesMinionNode.yaml#L90">https://github.com/openstack/murano-apps/blob/master/Docker/Kubernetes/KubernetesCluster/package/Classes/KubernetesMinionNode.yaml#L90</a></div><div><br></div><div>By that way as you can see there is cAdvisor setup on a new node too.</div><div><br></div><div>Thanks</div><div>Gosha</div>
<p class="MsoNormal" style="background-image:initial;background-repeat:initial"><span style="font-size:10.5pt;font-family:Calibri"></span></p>
</div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Apr 28, 2015 at 8:52 AM, Rabi Mishra <span dir="ltr"><<a href="mailto:ramishra@redhat.com" target="_blank">ramishra@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
----- Original Message -----<br>
> On Mon, Apr 27, 2015 at 12:28:01PM -0400, Rabi Mishra wrote:<br>
> > Hi All,<br>
> ><br>
> > Deploying Kubernetes(k8s) cluster on any OpenStack based cloud for<br>
> > container based workload is a standard deployment pattern. However,<br>
> > auto-scaling this cluster based on load would require some integration<br>
> > between k8s OpenStack components. While looking at the option of<br>
> > leveraging Heat ASG to achieve autoscaling, I came across few requirements<br>
> > that the list can discuss and arrive at the best possible solution.<br>
> ><br>
> > A typical k8s deployment scenario on OpenStack would be as below.<br>
> ><br>
> > - Master (single VM)<br>
> > - Minions/Nodes (AutoScalingGroup)<br>
> ><br>
> > AutoScaling of the cluster would involve both scaling of minions/nodes and<br>
> > scaling Pods(ReplicationControllers).<br>
> ><br>
> > 1. Scaling Nodes/Minions:<br>
> ><br>
> > We already have utilization stats collected at the hypervisor level, as<br>
> > ceilometer compute agent polls the local libvirt daemon to acquire<br>
> > performance data for the local instances/nodes.<br>
><br>
> I really doubts if those metrics are so useful to trigger a scaling<br>
> operation. My suspicion is based on two assumptions: 1) autoscaling<br>
> requests should come from the user application or service, not from the<br>
> controller plane, the application knows best whether scaling is needed;<br>
> 2) hypervisor level metrics may be misleading in some cases. For<br>
> example, it cannot give an accurate CPU utilization number in the case<br>
> of CPU overcommit which is a common practice.<br>
<br>
</div></div>I agree that correct utilization statistics is complex with virtual infrastructure.<br>
However, I think physical+hypervisor metrics (collected by compute agent) should be a<br>
good point to start.<br>
<span class=""><br>
> > Also, Kubelet (running on the node) collects the cAdvisor stats. However,<br>
> > cAdvisor stats are not fed back to the scheduler at present and scheduler<br>
> > uses a simple round-robin method for scheduling.<br>
><br>
> It looks like a multi-layer resource management problem which needs a<br>
> wholistic design. I'm not quite sure if scheduling at the container<br>
> layer alone can help improve resource utilization or not.<br>
<br>
</span>k8s scheduler is going to improve over time to use the cAdvisor/heapster metrics for<br>
better scheduling. IMO, we should leave that for k8s to handle.<br>
<br>
My point is on getting that metrics to ceilometer either from the nodes or from the \<br>
scheduler/master.<br>
<span class=""><br>
> > Req 1: We would need a way to push stats from the kubelet/cAdvisor to<br>
> > ceilometer directly or via the master(using heapster). Alarms based on<br>
> > these stats can then be used to scale up/down the ASG.<br>
><br>
> To send a sample to ceilometer for triggering autoscaling, we will need<br>
> some user credentials to authenticate with keystone (even with trusts).<br>
> We need to pass the project-id in and out so that ceilometer will know<br>
> the correct scope for evaluation. We also need a standard way to tag<br>
> samples with the stack ID and maybe also the ASG ID. I'd love to see<br>
> this done transparently, i.e. no matching_metadata or query confusions.<br>
><br>
> > There is an existing blueprint[1] for an inspector implementation for<br>
> > docker hypervisor(nova-docker). However, we would probably require an<br>
> > agent running on the nodes or master and send the cAdvisor or heapster<br>
> > stats to ceilometer. I've seen some discussions on possibility of<br>
> > leveraging keystone trusts with ceilometer client.<br>
><br>
> An agent is needed, definitely.<br>
><br>
> > Req 2: Autoscaling Group is expected to notify the master that a new node<br>
> > has been added/removed. Before removing a node the master/scheduler has to<br>
> > mark node as<br>
> > unschedulable.<br>
><br>
> A little bit confused here ... are we scaling the containers or the<br>
> nodes or both?<br>
<br>
</span>We would only focusing on the nodes. However, adding/removing nodes without the k8s master/scheduler<br>
knowing about it (so that it can schedule pods or make them unschedulable)would be useless.<br>
<span class=""><br>
> > Req 3: Notify containers/pods that the node would be removed for them to<br>
> > stop accepting any traffic, persist data. It would also require a cooldown<br>
> > period before the node removal.<br>
><br>
> There have been some discussions on sending messages, but so far I don't<br>
> think there is a conclusion on the generic solution.<br>
><br>
> Just my $0.02.<br>
<br>
</span>Thanks Qiming.<br>
<span class=""><br>
> BTW, we have been looking into similar problems in the Senlin project.<br>
<br>
</span>Great. We can probably discuss these during the Summit? I assume there is already a session<br>
on Senlin planned, right?<br>
<div class="HOEnZb"><div class="h5"><br>
><br>
> Regards,<br>
> Qiming<br>
><br>
> > Both requirement 2 and 3 would probably require generating scaling event<br>
> > notifications/signals for master and containers to consume and probably<br>
> > some ASG lifecycle hooks.<br>
> ><br>
> ><br>
> > Req 4: In case of too many 'pending' pods to be scheduled, scheduler would<br>
> > signal ASG to scale up. This is similar to Req 1.<br>
> ><br>
> ><br>
> > 2. Scaling Pods<br>
> ><br>
> > Currently manual scaling of pods is possible by resizing<br>
> > ReplicationControllers. k8s community is working on an abstraction,<br>
> > AutoScaler[2] on top of ReplicationController(RC) that provides<br>
> > intention/rule based autoscaling. There would be a requirement to collect<br>
> > cAdvisor/Heapster stats to signal the AutoScaler too. Probably this is<br>
> > beyond the scope of OpenStack.<br>
> ><br>
> > Any thoughts and ideas on how to realize this use-case would be<br>
> > appreciated.<br>
> ><br>
> ><br>
> > [1]<br>
> > <a href="https://review.openstack.org/gitweb?p=openstack%2Fceilometer-specs.git;a=commitdiff;h=6ea7026b754563e18014a32e16ad954c86bd8d6b" target="_blank">https://review.openstack.org/gitweb?p=openstack%2Fceilometer-specs.git;a=commitdiff;h=6ea7026b754563e18014a32e16ad954c86bd8d6b</a><br>
> > [2]<br>
> > <a href="https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/proposals/autoscaling.md" target="_blank">https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/proposals/autoscaling.md</a><br>
> ><br>
> > Regards,<br>
> > Rabi Mishra<br>
> ><br>
> ><br>
> > __________________________________________________________________________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><div dir="ltr"><font color="#999999"><span style="background-color:rgb(255,255,255)">Georgy Okrokvertskhov<br>
Architect,<br><span style="font-family:arial;font-size:small">OpenStack Platform Products,</span><br>
Mirantis</span><br>
<a href="http://www.mirantis.com/" target="_blank">http://www.mirantis.com</a><br>
Tel. +1 650 963 9828<br>
Mob. +1 650 996 3284</font><br></div></div>
</div>