<html><body><p>Hi Ricardo,<br> Great to have feedback from real use case. Spyros and I had a discussion on this in Austin <br>and we sketched out the implementation. Once you open the blueprint, we will add the details <br>and consider additional scenarios.<br>Ton,<br><br><img width="16" height="16" src="cid:1__=07BB0A9ADF8828218f9e8a93df938690918c07B@" border="0" alt="Inactive hide details for Ricardo Rocha ---08/07/2016 12:59:30 PM---Hi Ton. I think we should. Also in cases where multiple vol"><font color="#424282">Ricardo Rocha ---08/07/2016 12:59:30 PM---Hi Ton. I think we should. Also in cases where multiple volume types are available</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Ricardo Rocha <rocha.porto@gmail.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org></font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">08/07/2016 12:59 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [openstack-dev] [magnum] 2 million requests / sec, 100s of nodes</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><font size="4">Hi Ton.</font><br><br><font size="4">I think we should. Also in cases where multiple volume types are available (in our case with different iops) there would be additional parameters required to select the volume type. I'll add it this week.</font><br><br><font size="4">It's a detail though, spawning container clusters with Magnum is now super easy (and fast!).</font><br><br><font size="4">Cheers,</font><br><font size="4"> Ricardo</font><br><br><font size="4">On Fri, Aug 5, 2016 at 5:11 PM, Ton Ngo <</font><a href="mailto:ton@us.ibm.com" target="_blank"><u><font size="4" color="#0000FF">ton@us.ibm.com</font></u></a><font size="4">> wrote:</font><ul><font size="4">Hi Ricardo,<br>For your question 1, you can modify the Heat template to not create the Cinder volume and tweak the call to<br>configure-docker-storage.sh to use local storage. It should be fairly straightforward. You just need to make <br>sure the local storage of the flavor is sufficient to host the containers in the benchmark.<br>If you think this is a common scenario, we can open a blueprint for this option.<br>Ton,<br><br></font><img src="cid:1__=07BB0A9ADF8828218f9e8a93df938690918c07B@" width="16" height="16" alt="Inactive hide details for Ricardo Rocha ---08/05/2016 04:51:55 AM---Hi. Quick update is 1000 nodes and 7 million reqs/sec :) - "><font size="4" color="#424282">Ricardo Rocha ---08/05/2016 04:51:55 AM---Hi. Quick update is 1000 nodes and 7 million reqs/sec :) - and the number of</font><font size="4"><br></font><font color="#5F5F5F"><br>From: </font>Ricardo Rocha <<a href="mailto:rocha.porto@gmail.com" target="_blank"><u><font color="#0000FF">rocha.porto@gmail.com</font></u></a>><font color="#5F5F5F"><br>To: </font>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font color="#0000FF">openstack-dev@lists.openstack.org</font></u></a>><font color="#5F5F5F"><br>Date: </font>08/05/2016 04:51 AM
<p><font color="#5F5F5F"><br>Subject: </font>Re: [openstack-dev] [magnum] 2 million requests / sec, 100s of nodes<font size="4"><br></font><hr width="100%" size="2" align="left" noshade><font size="4"><br><br></font><font size="5"><br>Hi.</font><font size="4"><br></font><font size="5"><br>Quick update is 1000 nodes and 7 million reqs/sec :) - and the number of requests should be higher but we had some internal issues. We have a submission for barcelona to provide a lot more details.</font><font size="4"><br></font><font size="5"><br>But a couple questions came during the exercise:</font><font size="4"><br></font><font size="5"><br>1. Do we really need a volume in the VMs? On large clusters this is a burden, and local storage only should be enough?</font><font size="4"><br></font><font size="5"><br>2. We observe a significant delay (~10min, which is half the total time to deploy the cluster) on heat when it seems to be crunching the kube_minions nested stacks. Once it's done, it still adds new stacks gradually, so it doesn't look like it precomputed all the info in advance</font><font size="4"><br></font><font size="5"><br>Anyone tried to scale Heat to stacks this size? We end up with a stack with:<br>* 1000 nested stacks (depth 2)<br>* 22000 resources<br>* 47008 events</font><font size="4"><br></font><font size="5"><br>And already changed most of the timeout/retrial values for rpc to get this working.</font><font size="4"><br></font><font size="5"><br>This delay is already visible in clusters of 512 nodes, but 40% of the time in 1000 nodes seems like something we could improve. Any hints on Heat configuration optimizations for large stacks very welcome.</font><font size="4"><br></font><font size="5"><br>Cheers,<br> Ricardo</font><font size="4"><br></font><font size="5"><br>On Sun, Jun 19, 2016 at 10:59 PM, Brad Topol <</font><a href="mailto:btopol@us.ibm.com" target="_blank"><u><font size="5" color="#0000FF">btopol@us.ibm.com</font></u></a><font size="5">> wrote:</font><ul><ul><font size="5">Thanks Ricardo! This is very exciting progress!<br><br>--Brad<br><br><br>Brad Topol, Ph.D.<br>IBM Distinguished Engineer<br>OpenStack<br>(919) 543-0646<br>Internet: </font><a href="mailto:btopol@us.ibm.com" target="_blank"><u><font size="5" color="#0000FF">btopol@us.ibm.com</font></u></a><font size="5"><br>Assistant: Kendra Witherspoon (919) 254-0680<br></font><font size="4"><br></font><img src="cid:1__=07BB0A9ADF8828218f9e8a93df938690918c07B@" width="16" height="16" alt="Inactive hide details for Ton Ngo---06/17/2016 12:10:33 PM---Thanks Ricardo for sharing the data, this is really encouraging! T"><font size="5" color="#424282">Ton Ngo---06/17/2016 12:10:33 PM---Thanks Ricardo for sharing the data, this is really encouraging! Ton,</font><font size="4" color="#5F5F5F"><br><br>From: </font><font size="4">Ton Ngo/Watson/IBM@IBMUS</font><font size="4" color="#5F5F5F"><br>To: </font><font size="4">"OpenStack Development Mailing List \(not for usage questions\)" <</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="4" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="4">></font><font size="4" color="#5F5F5F"><br>Date: </font><font size="4">06/17/2016 12:10 PM</font><font size="4" color="#5F5F5F"><br>Subject: </font><font size="4">Re: [openstack-dev] [magnum] 2 million requests / sec, 100s of nodes </font><p><p><hr width="100%" size="2" align="left" noshade><font size="5"><br></font><font size="6"><br><br>Thanks Ricardo for sharing the data, this is really encouraging!<br>Ton,</font><font size="5"><br></font><font size="4"><br></font><img src="cid:1__=07BB0A9ADF8828218f9e8a93df938690918c07B@" width="16" height="16" alt="Inactive hide details for Ricardo Rocha ---06/17/2016 08:16:15 AM---Hi. Just thought the Magnum team would be happy to hear :)"><font size="6" color="#424282">Ricardo Rocha ---06/17/2016 08:16:15 AM---Hi. Just thought the Magnum team would be happy to hear :)</font><font size="5" color="#5F5F5F"><br><br>From: </font><font size="5">Ricardo Rocha <</font><a href="mailto:rocha.porto@gmail.com" target="_blank"><u><font size="5" color="#0000FF">rocha.porto@gmail.com</font></u></a><font size="5">></font><font size="5" color="#5F5F5F"><br>To: </font><font size="5">"OpenStack Development Mailing List (not for usage questions)" <</font><a href="mailto:openstack-dev@lists.openstack.org" target="_blank"><u><font size="5" color="#0000FF">openstack-dev@lists.openstack.org</font></u></a><font size="5">></font><font size="5" color="#5F5F5F"><br>Date: </font><font size="5">06/17/2016 08:16 AM</font><font size="5" color="#5F5F5F"><br>Subject: </font><font size="5">[openstack-dev] [magnum] 2 million requests / sec, 100s of nodes</font><font size="4"><br></font><hr width="100%" size="2" align="left" noshade><tt><font size="6"><br><br><br>Hi.<br><br>Just thought the Magnum team would be happy to hear :)<br><br>We had access to some hardware the last couple days, and tried some<br>tests with Magnum and Kubernetes - following an original blog post<br>from the kubernetes team.<br><br>Got a 200 node kubernetes bay (800 cores) reaching 2 million requests / sec.<br><br>Check here for some details:</font></tt><u><font size="4" color="#0000FF"><br></font></u><a href="https://openstack-in-production.blogspot.ch/2016/06/scaling-magnum-and-kubernetes-2-million.html" target="_blank"><tt><u><font size="6" color="#0000FF">https://openstack-in-production.blogspot.ch/2016/06/scaling-magnum-and-kubernetes-2-million.html</font></u></tt></a><tt><font size="6"><br><br>We'll try bigger in a couple weeks, also using the Rally work from<br>Winnie, Ton and Spyros to see where it breaks. Already identified a<br>couple issues, will add bugs or push patches for those. If you have<br>ideas or suggestions for the next tests let us know.<br><br>Magnum is looking pretty good!<br><br>Cheers,<br>Ricardo<br><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: </font></tt><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><tt><u><font size="6" color="#0000FF">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></tt></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><tt><u><font size="6" color="#0000FF">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></tt></a><font size="6"><br><br></font><tt><font size="5"><br><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: </font></tt><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><tt><u><font size="5" color="#0000FF">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></tt></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><tt><u><font size="5" color="#0000FF">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></tt></a><font size="5"><br><br></font><font size="4"><br></font><font size="5"><br><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: </font><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><u><font size="5" color="#0000FF">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><u><font size="5" color="#0000FF">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></a></ul></ul><tt><font size="4">__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: </font></tt><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><tt><u><font size="4" color="#0000FF">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></tt></a><tt><u><font size="4" color="#0000FF"><br></font></u></tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><tt><u><font size="4" color="#0000FF">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></tt></a><font size="4"><br><br><br></font><br><font size="4"><br>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: </font><a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" target="_blank"><u><font size="4" color="#0000FF">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</font></u></a><u><font size="4" color="#0000FF"><br></font></u><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank"><u><font size="4" color="#0000FF">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</font></u></a><font size="4"><br></font></ul><tt>__________________________________________________________________________<br>OpenStack Development Mailing List (not for usage questions)<br>Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br></tt><tt><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></tt><tt><br></tt><br><br><BR>
</body></html>