Sandy,<div><br></div><div>This is exactly what I was looking for, thanks for sending it along. I am glad to see that you all are thinking about this and have some targets in mind.</div><div><br></div><div>This weekend I will pull your numbers into a wiki page so we can begin to expand and elaborate on other scale metrics around Nova. In general I think they look like reasonable numbers although the number of hosts per zone looks a little low to me and the response time of 3 seconds seems a little generous ;-)</div>
<div><br></div><div>Thanks again!</div><div><br></div><div>-Blake</div><div><br><br><div class="gmail_quote">On Thu, Jan 26, 2012 at 10:40 AM, Sandy Walsh <span dir="ltr"><<a href="mailto:sandy.walsh@rackspace.com">sandy.walsh@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-size:10pt;font-family:Courier New">
Thanks Blake ... all very valid points.<br>
<br>
Based on our discussions yesterday (the ink is still wet on the whiteboard) we've been kicking around numbers in the following ranges:<br>
<br>
500-1000 hosts per zone (zone = single nova deployment. 1 db, 1 rabbit)<br>
25-100 instances per host (minimum flavor)<br>
3s api response time fully loaded (over that would be considered a failure). 'nova list' being the command that can bring down the house. But also 'nova boot' is another concern. We're always trying to get more async operations in there.<br>
<br>
Hosts per zone is a tricky one because we run into so many issues around network architecture, so your mileage may vary. Network is the limiting factor in this regard.<br>
<br>
All of our design decisions are being made with these metrics in mind. <br>
<br>
That said, we'd love to get more feedback on realistic metric expectations to ensure we're in the right church.<br>
<br>
Hope this is what you're looking for?<br>
<br>
-S<br>
<br>
<br>
<div style="font-size:16px;font-family:Times New Roman">
<hr>
<div style="direction:ltr"><font color="#000000" face="Tahoma"><b>From:</b> Blake Yeager [<a href="mailto:blake.yeager@gmail.com" target="_blank">blake.yeager@gmail.com</a>]<br>
<b>Sent:</b> Thursday, January 26, 2012 12:13 PM<br>
<b>To:</b> Sandy Walsh<br>
<b>Cc:</b> <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
<b>Subject:</b> Re: [Openstack] [Scaling][Orchestration] Zone changes. WAS: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts<br>
</font><br>
</div><div><div class="h5">
<div></div>
<div>Sandy,
<div><br>
</div>
<div>I am excited to hear about the work that is going on around communication between trusted zones and look forward to seeing what you have created.</div>
<div><br>
</div>
<div>In general, the scalability of Nova is an area where I think we need to put additional emphasis. Rackspace has done a lot of work on zones, but they don't seem to be receiving a lot of support from the rest of the community.</div>
<div><br>
</div>
<div>The OpenStack mission statement indicates the mission of the project is<strong style="line-height:18px;font-family:sans-serif">:</strong><font face="arial, helvetica, sans-serif"><span style="line-height:18px"> "</span><span style="line-height:18px"></span><span style="line-height:18px">To
produce the ubiquitous Open Source cloud computing platform that will meet the needs of public and private cloud providers regardless of size, by being simple to implement and massively scalable."</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px"><br>
</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px">I would challenge the community to ensure that scale is being given the appropriate focus in upcoming releases, especially Nova. Perhaps we
need to start by setting very specific scale targets for a single Nova zone in terms of nodes, instances, volumes, etc. I did a quick search of the wiki but I didn't find anything about scale targets. Does anyone know if something exists and I am just missing
it? Obviously scale will depend a lot on your specific hardware and configuration but we could start by saying with this minimum hardware spec and this configuration we want to be able to hit this scale. Likewise it would be nice to publish some statistics
about the scale that we believe a given release can operate at safely. This would tie into some of the QA/Testing work that Jay & team are working on.</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px"><br>
</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px">Does anyone have other thoughts about how we ensure we are all working toward building a massively scalable system?</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px"><br>
</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px">-Blake</span></font></div>
<div><font face="arial, helvetica, sans-serif"><span style="line-height:18px"><br>
</span></font></div>
<div>
<div class="gmail_quote">On Thu, Jan 26, 2012 at 9:20 AM, Sandy Walsh <span dir="ltr">
<<a href="mailto:sandy.walsh@rackspace.com" target="_blank">sandy.walsh@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Zones is going through some radical changes currently.<br>
<br>
Specifically, we're planning to use direct Rabbit-to-Rabbit communication between trusted Zones to avoid the complication of changes to OS API, Keystone and novaclient.<br>
<br>
To the user deploying Nova not much will change, there may be a new service to deploy (a Zones service), but that would be all. To a developer, the code in OS API will greatly simplify and the Distributed Scheduler will be able to focus on single zone scheduling
(vs doing both zone and host scheduling as it does today).<br>
<br>
We'll have more details soon, but we aren't planning on introducing the new stuff until we have a working replacement in place. The default Essex Scheduler now will largely be the same and the filters/weight functions will still carry forward, so any investments
there won't be lost.<br>
<br>
Stay tuned, we're hoping to get all this in a new blueprint soon.<br>
<br>
Hope it helps,<br>
Sandy<br>
<br>
________________________________________<br>
From: <a href="mailto:bounces@canonical.com" target="_blank">bounces@canonical.com</a> [<a href="mailto:bounces@canonical.com" target="_blank">bounces@canonical.com</a>] on behalf of Alejandro Comisario [<a href="mailto:question185840@answers.launchpad.net" target="_blank">question185840@answers.launchpad.net</a>]<br>
Sent: Thursday, January 26, 2012 8:50 AM<br>
To: Sandy Walsh<br>
Subject: Re: [Question #185840]: Multi-Zone finally working on ESSEX but cant "nova list" (KeyError: 'uuid') + doubts<br>
<br>
Question #185840 on OpenStack Compute (nova) changed:<br>
<a href="https://answers.launchpad.net/nova/+question/185840" target="_blank">https://answers.launchpad.net/nova/+question/185840</a><br>
<br>
Status: Answered => Open<br>
<br>
Alejandro Comisario is still having a problem:<br>
Sandy, Vish !<br>
<br>
Thanks for the replies ! let me get to the relevant points.<br>
<br>
#1 I totally agree with you guys, the policy for spawning instances<br>
maybe very special of each company strategy, but, as you can pass from<br>
"Fill First" to "Spread First" just adding a "reverse=True" on<br>
nova.scheduler.least_cost.weighted_sum" and<br>
"nova.scheduler.distributed_scheduler._schedule" maybe its a harmless<br>
addition to manipulate (since we are going to have a lot of zones across<br>
datacenters, and many different departments are going to create many<br>
instances to load-balance their applications, we really preffer<br>
SpreadFirst to make sure hight availability of the pools )<br>
<br>
#2 As we are going to test essex-3, i would like if you can tell me if<br>
the zones code from Chris Behrens is going to be added on Final Essex /<br>
Milestone 4, so we can keep testing other features, or you preffer us to<br>
load this as a bug to be fixed since maybe the code that broke is not<br>
going to have major changes.<br>
<br>
Kindest regards !<br>
<span><font color="#888888"><br>
--<br>
You received this question notification because you are a member of Nova<br>
Core, which is an answer contact for OpenStack Compute (nova).<br>
<br>
_______________________________________________<br>
Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
</font></span></blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</div>
</div>
</blockquote></div><br></div>