<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">
<br class="">
<div>
<blockquote type="cite" class="">
<div class="">On 26 Aug 2016, at 17:44, Andrew Laski <<a href="mailto:andrew@lascii.com" class="">andrew@lascii.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<title class=""></title>
<div class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">On Fri, Aug 26, 2016, at 11:01 AM, John Griffith wrote:<br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div style="font-family:monospace, monospace;" class=""><br class="">
</div>
<div class="">
<div class=""><br class="">
</div>
<div defang_data-gmailquote="yes" class="">
<div class="">On Fri, Aug 26, 2016 at 7:37 AM, Andrew Laski <span dir="ltr" class="">
<<a href="mailto:andrew@lascii.com" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">andrew@lascii.com</a>></span> wrote:<br class="">
</div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" defang_data-gmailquote="yes" class="">
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">On Fri, Aug 26, 2016, at 03:44 AM,<a href="mailto:Kostiantyn.Volenbovskyi@swisscom.com" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">Kostiantyn.Volenbovskyi@<wbr class="">swisscom.com</a><br class="">
</div>
<div class="">wrote:<br class="">
</div>
<div class=""><span class="">> Hi,<br class="">
> option 1 (=that's what patches suggest) sounds totally fine.<br class="">
> Option 3 > Allow block device mappings, when present, to mostly determine<br class="">
> instance packing<br class="">
> sounds like option 1+additional logic (=keyword 'mostly')<br class="">
> I think I miss to understand the part of 'undermining the purpose of the<br class="">
> flavor'<br class="">
> Why new behavior might require one more parameter to limit number of<br class="">
> instances of host?<br class="">
> Isn't it that those VMs will be under control of other flavor<br class="">
> constraints, such as CPU and RAM anyway and those will be the ones<br class="">
> controlling 'instance packing'?<br class="">
<br class="">
</span>Yes it is possible that CPU and RAM could be controlling instance</div>
<div class="">packing. But my understanding is that since those are often<br class="">
</div>
<div class="">oversubscribed<br class="">
</div>
</blockquote>
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class="">I don't understand why the oversubscription ratio matters here?<br class="">
</div>
<div style="font-family:monospace, monospace;display:inline;" class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">My experience is with environments where the oversubscription was used to be a little loose with how many vCPUs were allocated or how much RAM was allocated but disk was strictly controlled.</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">
<div defang_data-gmailquote="yes" class="">
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class=""> <br class="">
</div>
</div>
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class=""><br class="">
</div>
<div class=""> <br class="">
</div>
</div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" defang_data-gmailquote="yes" class="">
<div class="">while disk is not that it's actually the disk amounts<br class="">
</div>
<div class="">that control the packing on some environments.<br class="">
</div>
</blockquote>
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class="">Maybe an explanation of what you mean by "packing" here. Customers that I've worked with over the years have used CPU and Mem as their levers and the main thing that they care about in
terms of how many Instances go on a Node. I'd like to learn more about why that's wrong and that disk space is the mechanism that deployers use for this.<br class="">
</div>
</div>
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class=""><br class="">
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">By packing I just mean the various ways that different flavors fit on a host. A host may be designed to hold 1 xlarge, or 2 large, or 4 mediums, or 1 large and 2 mediums, etc... The challenge I see here is that the constraint can be managed by
using CPU or RAM or disk or some combination of the three. For deployers just using disk the above patches will change behavior for them.</div>
<div class=""><br class="">
</div>
<div class="">It's not wrong to use CPU/RAM, but it's not what everyone is doing. One purpose of this email was to gauge if it would be acceptable to only use CPU/RAM for packing.<br class="">
</div>
<div class=""><br class="">
</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">
<div defang_data-gmailquote="yes" class="">
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class=""><br class="">
</div>
<div class=""> <br class="">
</div>
</div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" defang_data-gmailquote="yes" class="">
<div class="">But that is a sub option<br class="">
</div>
<div class="">here, just document that disk amounts should not be used to determine<br class="">
</div>
<div class="">flavor packing on hosts and instead CPU and RAM must be used.<br class="">
</div>
<div class=""><span class=""><br class="">
> Does option 3 covers In case someone relied on eg. flavor root disk for<br class="">
> disk volume booted from volume - and now instance packing will change<br class="">
> once patches are implemented?<br class="">
<br class="">
</span>That's the goal. In a simple case of having hosts with 16 CPUs, 128GB of</div>
<div class="">RAM and 2TB of disk and a flavor with VCPU=4, RAM=32GB, root_gb=500GB,<br class="">
</div>
<div class="">swap/ephemeral=0 the deployer is stating that they want only 4 instances<br class="">
</div>
<div class="">on that host.<br class="">
</div>
</blockquote>
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class="">How do you arrive at that logic? What if they actually wanted a single VCPU=4,RAM=32GB,root_gb=500 but then they wanted the remaining resources split among Instances that were all 1 VCPU,
1 G ram and a 1 G root disk? <br class="">
</div>
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">My example assumes the one stated flavor. But if they have a smaller flavor then more than 4 instances would fit.</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">
<div defang_data-gmailquote="yes" class="">
<div class=""><br class="">
</div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" defang_data-gmailquote="yes" class="">
<div class="">If there is CPU and RAM oversubscription enabled then by<br class="">
</div>
<div class="">using volumes a user could end up with more than 4 instances on that<br class="">
</div>
<div class="">host. So a max_instances=4 setting could solve that. However I don't<br class="">
</div>
<div class="">like the idea of adding a new config, and I think it's too simplistic to<br class="">
</div>
<div class="">cover more complex use cases. But it's an option.<br class="">
</div>
</blockquote>
<div class="">
<div style="font-family:monospace, monospace;display:inline;" class=""><br class="">
</div>
</div>
<div style="font-family:monospace, monospace;" class="">I would venture to guess that most Operators would be sad to read that. So rather than give them an explicit lever that does exactly what they want clearly and explicitly we should make it as complex
as possible and have it be the result of a 4 or 5 variable equation? Not to mention it's completely dynamic (because it seems like
<br class="">
</div>
<div style="font-family:monospace, monospace;" class="">lots of clouds have more than one flavor).<br class="">
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">Is that lever exactly what they want? That's part of what I'd like to find out here. But currently it's possible to setup a situation where 1 large flavor or 4 small flavors fit on a host. So would the max_instances=4 setting be desired? Keeping
in mind that if the above patches merged 4 large flavors could be put on that host if they only use remote volumes and aren't using proper CPU/RAM limits.<br class="">
</div>
<div class=""><br class="">
</div>
<div class="">I probably was not clear enough in my original description or made some bad assumptions. The concern I have is that if someone is currently relying on disk sizes for their instance limits then the above patches change behavior for them and affect
capacity limits and planning. Is this okay and if not what do we do?</div>
<div class=""><br class="">
</div>
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>From a single operator perspective, we’d prefer an option which would allow boot from volume with a larger size than the flavour. The quota for volumes would avoid abuse.</div>
<div><br class="">
</div>
<div>The use cases we encounter are a standard set of flavors with defined core/memory/disk ratios which correspond to the underlying hardware (now SSD based so we are a little disk space constrained). A user wants to define a Windows VM which needs much more
disk space than the equivalent Linux flavour. We therefore suggest to use a volume to boot from. Given that we are disk constrained and want to maximise the CPU/memory usage, not using all the local space is less of an issue than asking them to choose a much
larger flavour which would lead to cores/memory being unused.</div>
<div><br class="">
</div>
<div>Tim</div>
<div><br class="">
</div>
<blockquote type="cite" class="">
<div class="">
<div class="">
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">
<div defang_data-gmailquote="yes" class="">
<div style="font-family:monospace, monospace;" class=""><br class="">
</div>
<div style="font-family:monospace, monospace;" class="">All I know is that the current state is broken. It's not just the scheduling problem, I could live with that probably since it's too hard to fix... but keep in mind that you're reporting the complete
wrong information for the Instance in these cases. My flavor says it's 5G, but in reality it's 200 or whatever. Rather than make it perfect we should just fix it. Personally I thought the proposals for a scheduler check and the addition of the Instances/Node
option was a win win for everyone. What am I <br class="">
</div>
<div style="font-family:monospace, monospace;" class="">missing? Would you rather a custom filter scheduler so it wasn't a config option? <br class="">
</div>
</div>
</div>
</div>
</blockquote>
<div class=""><br class="">
</div>
<div class="">There is another effort in progress to address the reporting issue. If you poke around Nova specs or conversations you'll hear it referred to as Resource Providers, though it's actually a series of specs with various names. There's certainly a
conversation that can be had about waiting for that effort vs trying to address resource tracking in a backportable manner, but that's not what I wanted to get into here.</div>
<div class=""><br class="">
</div>
<blockquote type="cite" class="">
<div dir="ltr" class="">
<div class="">
<div defang_data-gmailquote="yes" class="">
<div style="font-family:monospace, monospace;" class=""><br class="">
</div>
<blockquote style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204, 204, 204);padding-left:1ex;" defang_data-gmailquote="yes" class="">
<div class="">
<div class="">
<div class=""><br class="">
</div>
<div class="">><br class="">
</div>
<div class="">> BR,<br class="">
</div>
<div class="">> Konstantin<br class="">
</div>
<div class="">><br class="">
</div>
<div class="">> > -----Original Message-----<br class="">
</div>
<div class="">> > From: Andrew Laski [mailto:<a href="mailto:andrew@lascii.com" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">andrew@lascii.com</a>]<br class="">
</div>
<div class="">> > Sent: Thursday, August 25, 2016 10:20 PM<br class="">
</div>
<div class="">> > To: <a href="mailto:openstack-dev@lists.openstack.org" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
openstack-dev@lists.openstack.<wbr class="">org</a><br class="">
</div>
<div class="">> > Cc: <a href="mailto:openstack-operators@lists.openstack.org" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
openstack-operators@lists.<wbr class="">openstack.org</a><br class="">
</div>
<div class="">> > Subject: [Openstack-operators] [Nova] Reconciling flavors and block device<br class="">
</div>
<div class="">> > mappings<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Cross posting to gather some operator feedback.<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > There have been a couple of contentious patches gathering attention recently<br class="">
</div>
<div class="">> > about how to handle the case where a block device mapping supersedes flavor<br class="">
</div>
<div class="">> > information. Before moving forward on either of those I think we should have a<br class="">
</div>
<div class="">> > discussion about how best to handle the general case, and how to handle any<br class="">
</div>
<div class="">> > changes in behavior that results from that.<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > There are two cases presented:<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > 1. A user boots an instance using a Cinder volume as a root disk, however the<br class="">
</div>
<div class="">> > flavor specifies root_gb = x where x > 0. The current behavior in Nova is that the<br class="">
</div>
<div class="">> > scheduler is given the flavor root_gb info to take into account during scheduling.<br class="">
</div>
<div class="">> > This may disqualify some hosts from receiving the instance even though that disk<br class="">
</div>
<div class="">> > space is not necessary because the root disk is a remote volume.<br class="">
</div>
<div class="">> > <a href="https://review.openstack.org/#/c/200870/" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
https://review.openstack.org/#<wbr class="">/c/200870/</a><br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > 2. A user boots an instance and uses the block device mapping parameters to<br class="">
</div>
<div class="">> > specify a swap or ephemeral disk size that is less than specified on the flavor.<br class="">
</div>
<div class="">> > This leads to the same problem as above, the scheduler is provided information<br class="">
</div>
<div class="">> > that doesn't match the actual disk space to be consumed.<br class="">
</div>
<div class="">> > <a href="https://review.openstack.org/#/c/352522/" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
https://review.openstack.org/#<wbr class="">/c/352522/</a><br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Now the issue: while it's easy enough to provide proper information to the<br class="">
</div>
<div class="">> > scheduler on what the actual disk consumption will be when using block device<br class="">
</div>
<div class="">> > mappings that undermines one of the purposes of flavors which is to control<br class="">
</div>
<div class="">> > instance packing on hosts. So the outstanding question is to what extent should<br class="">
</div>
<div class="">> > users have the ability to use block device mappings to bypass flavor constraints?<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > One other thing to note is that while a flavor constrains how much local disk is<br class="">
</div>
<div class="">> > used it does not constrain volume size at all. So a user can specify an<br class="">
</div>
<div class="">> > ephemeral/swap disk <= to what the flavor provides but can have an arbitrary<br class="">
</div>
<div class="">> > sized root disk if it's a remote volume.<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Some possibilities:<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Completely allow block device mappings, when present, to determine instance<br class="">
</div>
<div class="">> > packing. This is what the patches above propose and there's a strong desire for<br class="">
</div>
<div class="">> > this behavior from some folks. But changes how many instances may fit on a<br class="">
</div>
<div class="">> > host which could be undesirable to some.<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Keep the status quo. It's clear that is undesirable based on the bug reports and<br class="">
</div>
<div class="">> > proposed patches above.<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Allow block device mappings, when present, to mostly determine instance<br class="">
</div>
<div class="">> > packing. By that I mean that the scheduler only takes into account local disk that<br class="">
</div>
<div class="">> > would be consumed, but we add additional configuration to Nova which limits<br class="">
</div>
<div class="">> > the number of instance that can be placed on a host. This is a compromise<br class="">
</div>
<div class="">> > solution but I fear that a single int value does not meet the needs of deployers<br class="">
</div>
<div class="">> > wishing to limit instances on a host. They want it to take into account cpu<br class="">
</div>
<div class="">> > allocations and ram and disk, in short a flavor :)<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > And of course there may be some other unconsidered solution. That's where<br class="">
</div>
<div class="">> > you, dear reader, come in.<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > Thoughts?<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > -Andrew<br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> ><br class="">
</div>
<div class="">> > ______________________________<wbr class="">_________________<br class="">
</div>
<div class="">> > OpenStack-operators mailing list<br class="">
</div>
<div class="">> > <a href="mailto:OpenStack-operators@lists.openstack.org" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
OpenStack-operators@lists.<wbr class="">openstack.org</a><br class="">
</div>
<div class="">> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
http://lists.openstack.org/<wbr class="">cgi-bin/mailman/listinfo/<wbr class="">openstack-operators</a><br class="">
</div>
<div class=""><br class="">
</div>
<div class="">______________________________<wbr class="">______________________________<wbr class="">______________<br class="">
</div>
<div class="">OpenStack Development Mailing List (not for usage questions)<br class="">
</div>
<div class="">Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">
OpenStack-dev-request@lists.<wbr class="">openstack.org?subject:<wbr class="">unsubscribe</a><br class="">
</div>
<div class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">http://lists.openstack.org/<wbr class="">cgi-bin/mailman/listinfo/<wbr class="">openstack-dev</a><br class="">
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
<div class=""><u class="">__________________________________________________________________________</u><br class="">
</div>
<div class="">OpenStack Development Mailing List (not for usage questions)<br class="">
</div>
<div class="">Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" class="">
OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br class="">
</div>
<div class=""><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" defang_rel="noreferrer" defang_data-ss1472225079="1" class="">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br class="">
</div>
</blockquote>
<div class=""><br class="">
</div>
</div>
_______________________________________________<br class="">
OpenStack-operators mailing list<br class="">
<a href="mailto:OpenStack-operators@lists.openstack.org" class="">OpenStack-operators@lists.openstack.org</a><br class="">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators<br class="">
</div>
</blockquote>
</div>
<br class="">
</body>
</html>