<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>I also just posted:</div>
<div><br>
</div>
<div><a href="https://review.openstack.org/#/c/26272">https://review.openstack.org/#/c/26272</a>/</div>
<div><br>
</div>
<div>That feature might be causing more DB traffic also…</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Matt Van Winkle <<a href="mailto:mvanwink@rackspace.com">mvanwink@rackspace.com</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 5, 2013 1:08 PM<br>
<span style="font-weight:bold">To: </span>Ryan Lane <<a href="mailto:rlane@wikimedia.org">rlane@wikimedia.org</a>><br>
<span style="font-weight:bold">Cc: </span>Paul Voccio <<a href="mailto:paul.voccio@rackspace.com">paul.voccio@rackspace.com</a>>, Rainya Mosher <<a href="mailto:rainya.mosher@rackspace.com">rainya.mosher@rackspace.com</a>>, "<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>"
<<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>>, Gabe Westmaas <<a href="mailto:gabe.westmaas@rackspace.com">gabe.westmaas@rackspace.com</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>It's about to, Ryan. I haven't been at my desk much since the discussion this morning, but I will get it bounced over shortly.</div>
<div><br>
</div>
<div>Thanks!</div>
<div>Matt</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ryan Lane <<a href="mailto:rlane@wikimedia.org">rlane@wikimedia.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, April 5, 2013 12:59 PM<br>
<span style="font-weight:bold">To: </span>Matt Van Winkle <<a href="mailto:mvanwink@rackspace.com">mvanwink@rackspace.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>" <<a href="mailto:user-committee@lists.openstack.org">user-committee@lists.openstack.org</a>>, Rainya Mosher <<a href="mailto:rainya.mosher@RACKSPACE.COM">rainya.mosher@RACKSPACE.COM</a>>,
Paul Voccio <<a href="mailto:paul.voccio@rackspace.com">paul.voccio@rackspace.com</a>>, Gabe Westmaas <<a href="mailto:gabe.westmaas@RACKSPACE.COM">gabe.westmaas@RACKSPACE.COM</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [User-committee] Feedback on Grizzly<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">On Fri, Apr 5, 2013 at 7:01 AM, Matt Van Winkle <span dir="ltr"><<a href="mailto:mvanwink@rackspace.com" target="_blank">mvanwink@rackspace.com</a>></span> wrote:<br>
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>Hello again, folks!</div>
<div><br>
</div>
<div>When I reached out a couple weeks ago, I mentioned that I was hoping that, along with being a large developer of OpenStack, Rackspace, could also contribute the committee's work as one of it's largest users via our public cloud. We just found our first
opportunity. This week we deployed an early release of Grizzly code to one of our data centers. </div>
<div><br>
</div>
<div>Going in, we knew there were quite a few database migrations. As we studied them, however, they presented some challenges in the manner that they were executed. Using them as they were would have meant extended downtime for the databases given the size
of our production data (row counts, etc). That downtime is problematic since it translates to the Public APIs being unavailable – something we aim to impact as minimally as possible during code deploys. Ultimately, we had to rewrite them ourselves to achieve
the same out comes with less DB unavailability. There is plenty of work the community can do, and the committee can help guide, around better ways to change database structure while maintaining as much uptime as possible. If you need more details, I'm happy
to bring the folks that worked on the rewrite into the conversation. Both will actually be at the summit.</div>
<div><br>
</div>
<div>The bigger surprise - and full disclosure, we learned a lot about the things we aren't testing in our deployment pipeline - was the dramatic increase in network traffic following the deploy. The new table structures, increased meta data and new queries
in this version translated to about 10X in the amount of data being returned for some queries. Add to that, the fact that compute nodes are regularly querying for certain information or often performing a "check in", and we saw a 3X (or more) increase in
network traffic on the management network we have for this particular DC (and it's a smaller one as our various deployments go). For now we have improved things slightly by turning off the following periodic tasks:</div>
<div><br>
</div>
<div>reboot_timeout</div>
<div>rescue_timeout</div>
<div>resize_confirm_window</div>
<div><br>
</div>
<div>These not running has the potential to create some other issues (zombies and such), but that can be managed.</div>
<div><br>
</div>
<div>It does look like the developers are already working on getting some of the queries updated:</div>
<div><br>
</div>
<div><a href="https://review.openstack.org/#/c/26136" target="_blank">https://review.openstack.org/#/c/26136</a>/</div>
<div><a href="https://review.openstack.org/#/c/26109" target="_blank">https://review.openstack.org/#/c/26109</a>/</div>
<div><br>
</div>
<div>All in all, I wanted to reach back out to you to follow up from before, because I think this particular experience is an excellent highlight that there is often a disconnect between some of the changes that come through to trunk and use of the code at
scale. Almost everyone who was dealt with the above will be in Oregon week after next, so I'm happy to drag any and all into the mix to discuss further.</div>
<div><br>
</div>
</div>
</blockquote>
<div><br>
</div>
<div></div>
</div>
</div>
<div class="gmail_extra" style="">Has this discussion been brought up with the developer community? I definitely feel it's important for the user committee to push on issues like this, but we should only push on topics that have already gone through normal
developer processes and aren't getting traction.</div>
<div class="gmail_extra" style=""><br>
</div>
<div class="gmail_extra" style="">- Ryan</div>
</div>
</div>
</div>
</span></div>
</div>
</span>
</body>
</html>