<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>You know Annie,</div>
<div>I thought through a lot of this, and you are right, there may be some things that can be easily built into nightly CI runs. Three thoughts I have:</div>
<div><br>
</div>
<div>1. One issue discussed in all this was the actual migrations themselves. All that would really be needed is some test databases with significant data – millions of rows, etc – to validate time/impact of migrations. The number of Dbs varry by particular
implementation, I know, but a start might be just a good sized cell and global DB to validate migrations for Nova code. We'd have to throw in a lot of dummy data. If not part of the CI job itself, perhaps jenkins jobs could at least be called to spin up
these DBs, restore data dumps and run migrations as need. IT would take some additional thought, but could be doable</div>
<div>2. As for the number of nodes, you have a very good point. We don't nee dthe hypervisor count as much as a ton of Vms. In the case of the traffic we saw in our implementation, it's because the compute nodes were contacting the Dbs. The compute nodes
are just Vms, so as you said, we could build a collection of machines running tons of "compute" nodes to test large traffic loads. Again, needs more thought, but we could take that somewhere useful</div>
<div>3. We actually use a small implementation of OpenStack to launch instances that become the control systems for our public cloud (APIs, Glance nodes, Dbs, etc…) so there is probably some learning there that could be leveraged in the above.</div>
<div><br>
</div>
<div>These are just some Saturday morning thoughts to the whole conversation, but yes, we might be able to build a few things into the community CI pipeline to help catch some of this stuff earlier.</div>
<div><br>
</div>
<div>Thanks!</div>
<div>Matt</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Annie Cheng <<a href="mailto:anniec@yahoo-inc.com">anniec@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 5, 2013 4:56 PM<br>
<span style="font-weight:bold">To: </span>Matt Van Winkle <<a href="mailto:mvanwink@rackspace.com">mvanwink@rackspace.com</a>>, "<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>" <<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>Perry Myers <<a href="mailto:pmyers@redhat.com">pmyers@redhat.com</a>>, "<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>" <<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>>,
Mark McLoughlin <<a href="mailto:markmc@redhat.com">markmc@redhat.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br>
</div>
<div>One idea regarding performance/scale lab that we are thinking about is whether there's possible to simulate thousands (or tens of thousands, depending on your scale) of hypervisor nodes with smaller set of physical nodes in order to test the control software
(openstack). In another words, if the purpose is to test openstack provisioning/deprovisioning software, do we really need real hypervisors (ie., we are not testing KVM, ZEN ..etc). If not, then do we even really need bare metal hosts? If not, then .. Can
we use openstack Vms to pretend its hypervisors? If we can do that, then how many nodes do we really need for a performance lab?</div>
<div><br>
</div>
<div>Once we figure our how many nodes we really need, can we ask for donation from our OpenStack family who care about even further success (stability, reliability, scalability) of OpenStack?</div>
<div><br>
</div>
<div>For issue like scalability, the bug must stop at design. And if we can't catch it at design, at least to see if we can catch it once everyday when a nightly perf test CI runs. Finding issue out when a release is done like you did, is a bit too late into
the game. Especially when we more and more end users (customers) who depend on OpenStack working 24x7. A release having serious scalability issue for some company may mean a need to skip entire release all together and not getting the benefit of great features
provided in a release.</div>
<div><br>
</div>
<div>Thanks!</div>
<div><br>
</div>
<div>Annie</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Matt Van Winkle <<a href="mailto:mvanwink@rackspace.com">mvanwink@rackspace.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 5, 2013 3:42 PM<br>
<span style="font-weight:bold">To: </span>Annie Cheng <<a href="mailto:anniec@yahoo-inc.com">anniec@yahoo-inc.com</a>>, "<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>" <<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>Perry Myers <<a href="mailto:pmyers@redhat.com">pmyers@redhat.com</a>>, "<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>" <<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>>,
Mark McLoughlin <<a href="mailto:markmc@redhat.com">markmc@redhat.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Thanks, Annie!</div>
<div><br>
</div>
<div>I'm not sure if I'm on the mailing list or can just send to it. I got your direct replies, but the didn't seem to be via the list. </div>
<div><br>
</div>
<div>As discussed, I've passed on the original note to the devs.</div>
<div><br>
</div>
<div>To your request, I haven't been able to get with the folks that did a lot of the migration overhaul, but I will and see what all we can pass along. I should add that we also preemptively purged a lot of data (deleted instances from the instance table)
specifically because of the migrations in an effort to further reduce run time.</div>
<div><br>
</div>
<div>As for the performance at scale lab, I love it! Might be a bit hard to find. What "scale" really means when it comes to hardware may be more than any entity wants to invest. I know we already struggle somewhat with our QE/Staging type environment sizes
versus production when it comes to certain things. As we grow, there may not be a solid way to really test "at scale" outside production. Like I said, though, we learned a lot about the things we weren't watching (Load Balancer traffic on control plane infrastructure)
in our development pipeline environments. While they are no where near the same scale as productions, we are already findings ways to measure, extrapolate and predict shifts in traffic with future releases so we don't run headlong into this again.</div>
<div><br>
</div>
<div>Thanks!</div>
<div>Matt</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Annie Cheng <<a href="mailto:anniec@yahoo-inc.com">anniec@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 5, 2013 9:38 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>" <<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>>, Matt Van Winkle <<a href="mailto:mvanwink@rackspace.com">mvanwink@rackspace.com</a>><br>
<span style="font-weight:bold">Cc: </span>Perry Myers <<a href="mailto:pmyers@redhat.com">pmyers@redhat.com</a>>, "<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>" <<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>>,
Mark McLoughlin <<a href="mailto:markmc@redhat.com">markmc@redhat.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Resent my previous two replies as now I just confirm my user-committee@ membership. Previous replies are stuck at the moderator queue. Hopefully the email will now go through. </div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Annie Cheng <<a href="mailto:anniec@yahoo-inc.com">anniec@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>" <<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>>, Annie Cheng <<a href="mailto:anniec@yahoo-inc.com">anniec@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 5, 2013 8:27 AM<br>
<span style="font-weight:bold">To: </span>"<a href="mailto:mvnwink@rackspace.com">mvnwink@rackspace.com</a>" <<a href="mailto:mvnwink@rackspace.com">mvnwink@rackspace.com</a>>, "<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>"
<<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>" <<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>>, Perry Myers <<a href="mailto:pmyers@redhat.com">pmyers@redhat.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Calibri; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="font-size: 14px; font-family: Calibri, sans-serif; ">
<div><br class="Apple-interchange-newline">
Another important topic that you point out we share the same concern is the community's awareness of upgrade path and downtime needed during an upgrade. As OpenStack mature, our end users (the users who uses VM) have higher expectation of SLA on API availability
uptime. In case of elasticity use case where users are expecting to increase/decrease capacity by provision/deprovision vms based on traffic/needs, API uptime is extremely important. </div>
<div><br>
</div>
<div>If you can share more details on DB migration to the community (ie., what was wrong, what's the fix) I think it will be a great learning experience for the community. For the work you've done on DB migration, do you plan to contribute your code upstream?</div>
<div><br>
<div>Thanks!</div>
<div><br>
</div>
<div>Annie</div>
<div><br>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family: Calibri; font-size: 11pt; text-align: left; color: black; border-bottom-width: medium; border-bottom-style: none; border-bottom-color: initial; border-left-width: medium; border-left-style: none; border-left-color: initial; padding-bottom: 0in; padding-left: 0in; padding-right: 0in; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; border-top-style: solid; border-right-width: medium; border-right-style: none; border-right-color: initial; padding-top: 3pt; ">
<span style="font-weight: bold; ">From:<span class="Apple-converted-space"> </span></span>Annie Cheng <<a href="mailto:anniec@yahoo-inc.com">anniec@yahoo-inc.com</a>><br>
<span style="font-weight: bold; ">Date:<span class="Apple-converted-space"> </span></span>Friday, April 5, 2013 8:10 AM<br>
<span style="font-weight: bold; ">To:<span class="Apple-converted-space"> </span></span>"<a href="mailto:mvnwink@rackspace.com">mvnwink@rackspace.com</a>" <<a href="mailto:mvnwink@rackspace.com">mvnwink@rackspace.com</a>>, "<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>"
<<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>><br>
<span style="font-weight: bold; ">Cc:<span class="Apple-converted-space"> </span></span>"<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>" <<a href="mailto:openstack-dev@yahoo-inc.com">openstack-dev@yahoo-inc.com</a>>, Perry Myers
<<a href="mailto:pmyers@redhat.com">pmyers@redhat.com</a>><br>
<span style="font-weight: bold; ">Subject:<span class="Apple-converted-space"> </span></span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Hi Matt,</div>
<div><br>
</div>
<div>>All in all, I wanted to reach back out to you to follow up from before, because I think this particular experience is an excellent highlight that there is often a disconnect between some of the changes that come through to trunk and use of the code at
scale.</div>
<div><br>
</div>
<div>Very interested in what your learning and can certainly feel your pain points from our experience here also. Yahoo! Is another company that will be deploying Grizzly at scale. My colleagues and I at Yahoo! love to get together with whoever else you can
drag into the room to learn about your experience. More importantly, whether the user-committee can help us drive the scale requirement into design (or during design and implementation, be more aware of scale impact). </div>
<div><br>
</div>
<div>Another interesting topic would be whether there can be a performance/scale lab available/set up for the community, where nightly, trunk code can be launched and results can be published, so we can catch perf/scale issue early rather than late. This is
quite a standard practice to have perf/scale lab at large scale companies like Yahoo!, as OpenStack matures and more widely adopted to larger scale operators, is this something we can consider for the reliability and scalability of OpenStack releases moving
forward.</div>
<div><br>
</div>
<div>Annie</div>
</div>
</div>
</span></div>
</span></span></span>
<div><br>
</div>
<div><br>
</div>
<div>
<blockquote type="cite" style="font-family: Calibri; font-size: medium; ">
<div><b>From:</b> Matt Van Winkle <<a href="mailto:mvanwink@rackspace.com">mvanwink@rackspace.com</a>><br>
<b>Date:</b> April 5, 2013, 7:01:26 AM PDT<br>
<b>To:</b> "<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>" <<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>><br>
<b>Cc:</b> Rainya Mosher <<a href="mailto:rainya.mosher@rackspace.com">rainya.mosher@rackspace.com</a>>, Paul Voccio <<a href="mailto:paul.voccio@rackspace.com">paul.voccio@rackspace.com</a>>, Gabe Westmaas <<a href="mailto:gabe.westmaas@rackspace.com">gabe.westmaas@rackspace.com</a>><br>
<b>Subject:</b> <b>[User-committee] Feedback on Grizzly</b><br>
<br>
</div>
</blockquote>
<blockquote type="cite" style="font-family: Calibri; font-size: medium; ">
<div>
<div>Hello again, folks!</div>
<div><br>
</div>
<div>When I reached out a couple weeks ago, I mentioned that I was hoping that, along with being a large developer of OpenStack, Rackspace, could also contribute the committee's work as one of it's largest users via our public cloud. We just found our first
opportunity. This week we deployed an early release of Grizzly code to one of our data centers. </div>
<div><br>
</div>
<div>Going in, we knew there were quite a few database migrations. As we studied them, however, they presented some challenges in the manner that they were executed. Using them as they were would have meant extended downtime for the databases given the size
of our production data (row counts, etc). That downtime is problematic since it translates to the Public APIs being unavailable – something we aim to impact as minimally as possible during code deploys. Ultimately, we had to rewrite them ourselves to achieve
the same out comes with less DB unavailability. There is plenty of work the community can do, and the committee can help guide, around better ways to change database structure while maintaining as much uptime as possible. If you need more details, I'm happy
to bring the folks that worked on the rewrite into the conversation. Both will actually be at the summit.</div>
<div><br>
</div>
<div>The bigger surprise - and full disclosure, we learned a lot about the things we aren't testing in our deployment pipeline - was the dramatic increase in network traffic following the deploy. The new table structures, increased meta data and new queries
in this version translated to about 10X in the amount of data being returned for some queries. Add to that, the fact that compute nodes are regularly querying for certain information or often performing a "check in", and we saw a 3X (or more) increase in
network traffic on the management network we have for this particular DC (and it's a smaller one as our various deployments go). For now we have improved things slightly by turning off the following periodic tasks:</div>
<div><br>
</div>
<div>reboot_timeout</div>
<div>rescue_timeout</div>
<div>resize_confirm_window</div>
<div><br>
</div>
<div>These not running has the potential to create some other issues (zombies and such), but that can be managed.</div>
<div><br>
</div>
<div>It does look like the developers are already working on getting some of the queries updated:</div>
<div><br>
</div>
<div><a href="https://review.openstack.org/#/c/26136">https://review.openstack.org/#/c/26136</a>/</div>
<div><a href="https://review.openstack.org/#/c/26109">https://review.openstack.org/#/c/26109</a>/</div>
<div><br>
</div>
<div>All in all, I wanted to reach back out to you to follow up from before, because I think this particular experience is an excellent highlight that there is often a disconnect between some of the changes that come through to trunk and use of the code at
scale. Almost everyone who was dealt with the above will be in Oregon week after next, so I'm happy to drag any and all into the mix to discuss further.</div>
<div><br>
</div>
<div>Thanks so much!</div>
<div>Matt</div>
</div>
</blockquote>
<blockquote type="cite" style="font-family: Calibri; font-size: medium; ">
<div><span>_______________________________________________</span><br>
<span>User-committee mailing list</span><br>
<span><a href="mailto:User-committee@lists.openstack.org">User-committee@lists.openstack.org</a></span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee">http://lists.openstack.org/cgi-bin/mailman/listinfo/user-committee</a></span></div>
</blockquote>
</div>
</div>
</div>
</span></div>
</div>
</span></div>
</div>
</span>
</body>
</html>