<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>Tim,</div>
<div><br>
</div>
<div>These are great topics to discuss, so far the only strong strategy that we have documented as community is to make a proper pre-production sizing of your Cloud and your Instances flavors which is not enough to not end up in that tetris problem that you
mentioned before.</div>
<div><br>
</div>
<div>I second this proposal!</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Tim Bell <<a href="mailto:Tim.Bell@cern.ch">Tim.Bell@cern.ch</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, January 15, 2015 at 10:24 PM<br>
<span style="font-weight:bold">To: </span>matt <<a href="mailto:matt@nycresistor.com">matt@nycresistor.com</a>>, George Shuklin <<a href="mailto:george.shuklin@gmail.com">george.shuklin@gmail.com</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] Lets talk capacity monitoring<br>
</div>
<div><br>
</div>
<div xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri",sans-serif;
color:#1F497D;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">One good topic to try to pin down at the Ops meet up would be how we could do the flavour/aggregate/project/hypervisor mappings. We’ve got a local
patch for some function but it was not possible to get the right way to do it agreed (<a href="https://blueprints.launchpad.net/nova/+spec/multi-tenancy-isolation-only-aggregates">https://blueprints.launchpad.net/nova/+spec/multi-tenancy-isolation-only-aggregates</a>).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">We do a fair amount of ‘just how many of flavour X could I accept’ but when we add in the various combinations of availability zones and cells, it
can be easy to run out in one corner of the cloud. There is also the tetris problem where one hypervisor has some memory left over, another one has some CPU but you can’t combine them (we’d need some hardware support to get that feature working…)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);">Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125);"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;">From:</span></b><span lang="EN-US" style="font-size: 11pt; font-family: Calibri, sans-serif;"> matt [<a href="mailto:matt@nycresistor.com">mailto:matt@nycresistor.com</a>]
<br>
<b>Sent:</b> 16 January 2015 00:30<br>
<b>To:</b> George Shuklin<br>
<b>Cc:</b> <a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a><br>
<b>Subject:</b> Re: [Openstack-operators] Lets talk capacity monitoring<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I've found histograms to be pretty useful in figuring out patterns during sizable time deltas... and anomaly detection there can highlight stuff you might want to check out ( ie raise the alert condition on that device ).<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">example of a histogram i did many many moons ago to track disk sizes from our nagios plugin that did dynamic disk free analytics. I don't have any of the animated GIFs I made that showed fluctuations over days... but that was great from
a human visual sense.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I suppose this could be further automated and refined, I've not been focused here anymore though.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">-Matt<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Jan 15, 2015 at 3:08 PM, George Shuklin <<a href="mailto:george.shuklin@gmail.com" target="_blank">george.shuklin@gmail.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal">On 01/15/2015 06:43 PM, Jesse Keating wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal" style="margin-bottom:12.0pt">We have a need to better manage the various openstack capacities across our numerous clouds. We want to be able to detect when capacity of one system or another is approaching the point where it would be a good
idea to arrange to increase that capacity. Be it volume space, VCPU capability, object storage space, etc...<br>
<br>
What systems are you folks using to monitor and react to such things?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
In our case we are using standard metrics (ganglia) and monitoring (shinken). I have thoughts about 'capacity planing', but the problem is that you cannot separate payload from wasted resources. For example, when snapshot is created, it eats space on compute
(for some configuration) beyond flavor limits. If instance boots, _base is used too (and if instance is booting from big snapshot, it use more space in _base, than in /instances). CPU can be heavily used by many host-internal processes, and memory is shared
with management software (which can be greedy too). IO can be overspend on snapshots/booting.<br>
<br>
So we are using cumulative graphs for free space, cpu usage, memory usage. It does not cover flavor/aggregate/pinning-to-host-by-metadata cases, but overall give some feeling about available free resources.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
_______________________________________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</span>
</body>
</html>