[openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource

Tim Hinrichs thinrichs at vmware.com
Wed Mar 5 16:49:29 UTC 2014 

Hi Gokul, 

Thanks for working out how all these policy initiatives relate to each other. I'll be spending some time diving into the ones I hadn't heard about. 

I made some additional comments about Congress below. 

Tim 

----- Original Message -----

From: "Jay Lau" <jay.lau.513 at gmail.com> 
To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org> 
Sent: Wednesday, March 5, 2014 7:31:55 AM 
Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage compute/storage resource 

Hi Gokul, 




2014-03-05 3:30 GMT+08:00 Gokul Kandiraju < gokul4open at gmail.com > : 





Dear All, 

We are working on a framework where we want to monitor the system and take certain actions when specific events or situations occur. Here are two examples of ‘different’ situations: 

Example 1: A VM’s-Owner and N/W’s-owner are different ==> this could mean a violation ==> we need to take some action 

Example 2: A simple policy such as (VM-migrate of all VMs on possible node failure) OR (a more complex Energy Policy that may involve optimization). 

Both these examples need monitoring and actions to be taken when certain events happen (or through polling). However, the first one falls into the Compliance domain with Boolean conditions getting evaluated while the second one may require a more richer set of expression allowing for sequences or algorithms. 





So far, based on this discussion, it seems that these are *separate* initiatives in the community. I am understanding the Congress project to be in the domain of Boolean conditions (used for Compliance, etc.) where as the Run-time-policies (Jay's proposal) where policies can be expressed as rules, algorithms with higher-level goals. Is this understanding correct? 

Also, looking at all the mails, this is what I am reading: 

1. Congress -- Focused on Compliance [ is that correct? ] (Boolean constraints and logic) 



[Tim] Your characterization of boolean constraints for Congress is probably a good one. Congress won't be solving optimization/numeric problems any time soon if ever. However, I could imagine that down the road we could tell Congress here's the policy (optimization or Boolean) that we want to enforce, and it would carve off say the Load-balancing part of the policy and send it to the Runtime-Policies component; or it would carve off the placement policy and send it to the SolverScheduler. Not saying I know how to do this today, but that's always been part of the goal for Congress: to have a central point for admins to control the global policy being enforced throughout the datacenter/cloud. 

The other delta here is that the Congress policy language is general-purpose, so there's not a list of policy types that it will handle (Load Balancing, Placement, Energy). That generality comes with a price: that Congress must rely on other enforcement points, such as the ones below, to handle complicated policy enforcement problems. 



<blockquote>



2. Runtime-Policies -- <Jay’s mail> -- Focused on Runtime policies for Load Balancing, Availability, Energy, etc. (sequences of actions, rules, algorithms) 

</blockquote>

[Jay] Yes, exactly. 

<blockquote>





3. SolverScheduler -- Focused on Placement [ static or runtime ] and will be invoked by the (above) policy engines 




4. Gantt – Focused on (Holistic) Scheduling 

</blockquote>

[Jay] For 3 and 4, I was always thinking Gantt is doing something for implementing SolverScheduler, not sure if run time policy can be included. 

<blockquote>





5. Neat -- seems to be a special case of Runtime-Policies (policies based on Load) 



Would this be correct understanding? We need to understand this to contribute to the right project. :) 



Thanks! 

-Gokul 


On Fri, Feb 28, 2014 at 5:46 PM, Jay Lau < jay.lau.513 at gmail.com > wrote: 

<blockquote>

Hi Yathiraj and Tim, 

Really appreciate your comments here ;-) 

I will prepare some detailed slides or documents before summit and we can have a review then. It would be great if OpenStack can provide "DRS" features. 

Thanks, 

Jay 



2014-03-01 6:00 GMT+08:00 Tim Hinrichs < thinrichs at vmware.com > : 


<blockquote>
Hi Jay, 

I think the Solver Scheduler is a better fit for your needs than Congress because you know what kinds of constraints and enforcement you want. I'm not sure this topic deserves an entire design session--maybe just talking a bit at the summit would suffice (I *think* I'll be attending). 

Tim 

----- Original Message ----- 
| From: "Jay Lau" < jay.lau.513 at gmail.com > 
| To: "OpenStack Development Mailing List (not for usage questions)" < openstack-dev at lists.openstack.org > 
| Sent: Wednesday, February 26, 2014 6:30:54 PM 
| Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal for OpenStack run time policy to manage 
| compute/storage resource 
| 
| 
| 
| 
| 
| 
| Hi Tim, 
| 
| I'm not sure if we can put resource monitor and adjust to 
| solver-scheduler (Gantt), but I have proposed this to Gantt design 
| [1], you can refer to [1] and search "jay-lau-513". 
| 
| IMHO, Congress does monitoring and also take actions, but the actions 
| seems mainly for adjusting single VM network or storage. It did not 
| consider migrating VM according to hypervisor load. 
| 
| Not sure if this topic deserved to be a design session for the coming 
| summit, but I will try to propose. 
| 
| 
| 
| 
| [1] https://etherpad.openstack.org/p/icehouse-external-scheduler 
| 
| 
| 
| Thanks, 
| 
| 
| Jay 
| 
| 
| 
| 2014-02-27 1:48 GMT+08:00 Tim Hinrichs < thinrichs at vmware.com > : 
| 
| 
| Hi Jay and Sylvain, 
| 
| The solver-scheduler sounds like a good fit to me as well. It clearly 
| provisions resources in accordance with policy. Does it monitor 
| those resources and adjust them if the system falls out of 
| compliance with the policy? 
| 
| I mentioned Congress for two reasons. (i) It does monitoring. (ii) 
| There was mention of compute, networking, and storage, and I 
| couldn't tell if the idea was for policy that spans OS components or 
| not. Congress was designed for policies spanning OS components. 
| 
| 
| Tim 
| 
| ----- Original Message ----- 
| 
| | From: "Jay Lau" < jay.lau.513 at gmail.com > 
| | To: "OpenStack Development Mailing List (not for usage questions)" 
| | < openstack-dev at lists.openstack.org > 
| 
| 
| | Sent: Tuesday, February 25, 2014 10:13:14 PM 
| | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A proposal 
| | for OpenStack run time policy to manage 
| | compute/storage resource 
| | 
| | 
| | 
| | 
| | 
| | Thanks Sylvain and Tim for the great sharing. 
| | 
| | @Tim, I also go through with Congress and have the same feeling 
| | with 
| | Sylvai, it is likely that Congress is doing something simliar with 
| | Gantt providing a holistic way for deploying. What I want to do is 
| | to provide some functions which is very similar with VMWare DRS 
| | that 
| | can do some adaptive scheduling automatically. 
| | 
| | @Sylvain, can you please show more detail for what "Pets vs. 
| | Cattles 
| | analogy" means? 
| | 
| | 
| | 
| | 
| | 2014-02-26 9:11 GMT+08:00 Sylvain Bauza < sylvain.bauza at gmail.com > 
| | : 
| | 
| | 
| | 
| | Hi Tim, 
| | 
| | 
| | As per I'm reading your design document, it sounds more likely 
| | related to something like Solver Scheduler subteam is trying to 
| | focus on, ie. intelligent agnostic resources placement on an 
| | holistic way [1] 
| | IIRC, Jay is more likely talking about adaptive scheduling 
| | decisions 
| | based on feedback with potential counter-measures that can be done 
| | for decreasing load and preserving QoS of nodes. 
| | 
| | 
| | That said, maybe I'm wrong ? 
| | 
| | 
| | [1] https://blueprints.launchpad.net/nova/+spec/solver-scheduler 
| | 
| | 
| | 
| | 2014-02-26 1:09 GMT+01:00 Tim Hinrichs < thinrichs at vmware.com > : 
| | 
| | 
| | 
| | 
| | Hi Jay, 
| | 
| | The Congress project aims to handle something similar to your use 
| | cases. I just sent a note to the ML with a Congress status update 
| | with the tag [Congress]. It includes links to our design docs. Let 
| | me know if you have trouble finding it or want to follow up. 
| | 
| | Tim 
| | 
| | 
| | 
| | ----- Original Message ----- 
| | | From: "Sylvain Bauza" < sylvain.bauza at gmail.com > 
| | | To: "OpenStack Development Mailing List (not for usage 
| | | questions)" 
| | | < openstack-dev at lists.openstack.org > 
| | | Sent: Tuesday, February 25, 2014 3:58:07 PM 
| | | Subject: Re: [openstack-dev] [OpenStack][Runtime Policy] A 
| | | proposal 
| | | for OpenStack run time policy to manage 
| | | compute/storage resource 
| | | 
| | | 
| | | 
| | | Hi Jay, 
| | | 
| | | 
| | | Currently, the Nova scheduler only acts upon user request (either 
| | | live migration or boot an instance). IMHO, that's something Gantt 
| | | should scope later on (or at least there could be some space 
| | | within 
| | | the Scheduler) so that Scheduler would be responsible for 
| | | managing 
| | | resources on a dynamic way. 
| | | 
| | | 
| | | I'm thinking of the Pets vs. Cattles analogy, and I definitely 
| | | think 
| | | that Compute resources could be treated like Pets, provided the 
| | | Scheduler does a move. 
| | | 
| | | 
| | | -Sylvain 
| | | 
| | | 
| | | 
| | | 2014-02-26 0:40 GMT+01:00 Jay Lau < jay.lau.513 at gmail.com > : 
| | | 
| | | 
| | | 
| | | 
| | | Greetings, 
| | | 
| | | 
| | | Here I want to bring up an old topic here and want to get some 
| | | input 
| | | from you experts. 
| | | 
| | | 
| | | Currently in nova and cinder, we only have some initial placement 
| | | polices to help customer deploy VM instance or create volume 
| | | storage 
| | | to a specified host, but after the VM or the volume was created, 
| | | there was no policy to monitor the hypervisors or the storage 
| | | servers to take some actions in the following case: 
| | | 
| | | 
| | | 1) Load Balance Policy: If the load of one server is too heavy, 
| | | then 
| | | probably we need to migrate some VMs from high load servers to 
| | | some 
| | | idle servers automatically to make sure the system resource usage 
| | | can be balanced. 
| | | 
| | | 2) HA Policy: If one server get down for some hardware failure or 
| | | whatever reasons, there is no policy to make sure the VMs can be 
| | | evacuated or live migrated (Make sure migrate the VM before 
| | | server 
| | | goes down) to other available servers to make sure customer 
| | | applications will not be affect too much. 
| | | 
| | | 3) Energy Saving Policy: If a single host load is lower than 
| | | configured threshold, then low down the frequency of the CPU to 
| | | save 
| | | energy; otherwise, increase the CPU frequency. If the average 
| | | load 
| | | is lower than configured threshold, then shutdown some 
| | | hypervisors 
| | | to save energy; otherwise, power on some hypervisors to load 
| | | balance. Before power off a hypervisor host, the energy policy 
| | | need 
| | | to live migrate all VMs on the hypervisor to other available 
| | | hypervisors; After Power on a hypervisor host, the Load Balance 
| | | Policy will help live migrate some VMs to the new powered 
| | | hypervisor. 
| | | 
| | | 4) Customized Policy: Customer can also define some customized 
| | | policies based on their specified requirement. 
| | | 
| | | 5) Some run-time policies for block storage or even network. 
| | | 
| | | 
| | | 
| | | I borrow the idea from VMWare DRS (Thanks VMWare DRS), and there 
| | | indeed many customers want such features. 
| | | 
| | | 
| | | 
| | | I have filed a bp here [1] long ago, but after some discussion 
| | | with 
| | | Russell, we think that this should not belong to nova but other 
| | | projects. Till now, I did not find a good place where we can put 
| | | this in, can any of you show some comments? 
| | | 
| | | 
| | | 
| | | [1] 
| | | https://blueprints.launchpad.net/nova/+spec/resource-optimization-service 
| | | 
| | | -- 
| | | 
| | | 
| | | Thanks, 
| | | 
| | | Jay 
| | | 
| | | _______________________________________________ 
| | | OpenStack-dev mailing list 
| | | OpenStack-dev at lists.openstack.org 
| | | http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
| | | 
| | | 
| | | 
| | | _______________________________________________ 
| | | OpenStack-dev mailing list 
| | | OpenStack-dev at lists.openstack.org 
| | | https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=XDB3hT4WE2iDrNVK0sQ8qKooX2r1T4E%2BVHek3GREhnE%3D%0A&s=e2346cd017c9d8108c12a101892492e2ac75953e4a5ea5c17394c775cf086d7f 
| | 
| | 
| | | 
| | 
| | _______________________________________________ 
| | OpenStack-dev mailing list 
| | OpenStack-dev at lists.openstack.org 
| | http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
| | 
| | 
| | _______________________________________________ 
| | OpenStack-dev mailing list 
| | OpenStack-dev at lists.openstack.org 
| | http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
| | 
| | 
| | 
| | 
| | -- 
| | 
| | 
| | Thanks, 
| | 
| | Jay 
| | 
| | _______________________________________________ 
| | OpenStack-dev mailing list 
| | OpenStack-dev at lists.openstack.org 
| | https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=E45fhmBClHHPExheGdRk0z%2Bj72gQAP4Zc1W3XElJx60%3D%0A&s=684cee6930f5d74f56e1ab9fc42e5f3c2511f07948f357040ca2dc175c4ccee6 
| 
| 
| | 
| 
| _______________________________________________ 
| OpenStack-dev mailing list 
| OpenStack-dev at lists.openstack.org 
| http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 
| 
| 
| 
| -- 
| 
| 
| Thanks, 
| 
| Jay 
| 
| _______________________________________________ 
| OpenStack-dev mailing list 
| OpenStack-dev at lists.openstack.org 
| https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=oXtNDrtlvCX0d%2BN7rnJXx5e3YjkX%2FGQHVTRP%2BN7hKrw%3D%0A&s=40f6f376ab93e166f327385661db38cf88ca6f2563fbc003cd94d81bf596f9c4 
| 

_______________________________________________ 
OpenStack-dev mailing list 
OpenStack-dev at lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 

</blockquote>




-- 
Thanks, 

Jay 

_______________________________________________ 
OpenStack-dev mailing list 
OpenStack-dev at lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 


</blockquote>



_______________________________________________ 
OpenStack-dev mailing list 
OpenStack-dev at lists.openstack.org 
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev 


</blockquote>




-- 
Thanks, 

Jay 

_______________________________________________ 
OpenStack-dev mailing list 
OpenStack-dev at lists.openstack.org 
https://urldefense.proofpoint.com/v1/url?u=http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev&k=oIvRg1%2BdGAgOoM1BIlLLqw%3D%3D%0A&r=%2FZ35AkRhp2kCW4Q3MPeE%2BxY2bqaf%2FKm29ZfiqAKXxeo%3D%0A&m=MTqbj4EaM%2BUfH6JKC0wjfFzgqMq7tnMIJEM9NQFgpyw%3D%0A&s=d09bc683f6810994e701463bee318385b8630d44065fa2b853b1b8a333e6ad68 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140305/6ff38a36/attachment.html>

More information about the OpenStack-dev mailing list