<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">As you mention, our current unit is HEP-Spec06 which is based on a subset of the Spec2006 benchmark suite.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">From the cloud accounting, we just get given the wall clock and the CPU time used. However, depending on the performance rating of
the machine, the value of this time slot to the physicist can differ significantly.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Finding out which hypervisor the guest is running during that time window is also not available (AFAIK) in the accounting record. So,
finding the underlying hardware is not possible from the pure accounting data (so an external mapping is needed such as the nova db)<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">We had thought about having some way that ceilometer would be informed about the rating of the machine and then report this as part
of the record to the accounting system which could then choose whether to scale the values as part of the accounting process.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Things get really complicated when you have potentially overcommitted or undercommitted hypervisors. In the case of overcommit, clearly
the vCPU is worth less than the reference benchmark. In the case of undercommit, the most recent CPUs can do various frequency scaling tricks.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Not an easy problem to solve, even in the simple case.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US">Tim<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D;mso-fareast-language:EN-US"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span lang="EN-US" style="font-size:11.0pt;font-family:"Calibri",sans-serif"> John Hearns [mailto:hearnsj@googlemail.com]
<br>
<b>Sent:</b> 14 January 2015 12:37<br>
<b>To:</b> openstack-hpc@lists.openstack.org<br>
<b>Subject:</b> Re: [openstack-hpc] How are you using OpenStack for your HPC workflow? (We're looking at Ironic!)<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Tim Bell wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">- Accounting - The cloud accounting systems are based on vCPUs. There is no concept in OpenStack of a relative performance measure so you could be allocated a VM on 3 year old hardware or the latest and the
vCPU metric is the same. In the high throughput use cases, there should be a relative unit which is used to scale with regards to the amount of work that could be done. With over 20 different hardware configurations running (competitive public procurement
cycles over 4 years), we can't define 100 flavors with different accounting rates and expect our users to guess which ones have capacity.</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">This is quite interesting. I guess that CERN do not use the VUP (Vax Unit of Power) any more!</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:10.0pt">Could you not use some sort of virtual currency, translating time occupied on a server X HEPSpec rating of that server into a virtual amount of money or units?</span><o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On 14 January 2015 at 06:24, Tim Bell <<a href="mailto:Tim.Bell@cern.ch" target="_blank">Tim.Bell@cern.ch</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0cm 0cm 0cm 6.0pt;margin-left:4.8pt;margin-right:0cm">
<p class="MsoNormal"><br>
At CERN, we have taken two approaches for allocating compute resources on a single cloud<br>
<br>
- Run classic high throughput computing using a batch system (LSF in our case) using virtualised resources. There are some losses from the memory needs of the hypervisor (so you have a slot or so less per machine) and some local I/O impacts but other than that,
there are major benefits of being able to automate recovery/recycling of VMs such as for security of hardware issues.<br>
<br>
- Run cloud services ala Amazon. Here users have one of the standard images such as CentOS or their own if they wish with cloud-init.<br>
<br>
Compute resources at CERN are allocated using a pledge system, i.e. the experiments request and justify their needs for the year, resources are purchased and then allocated out according to these pledges. There is no charging as such.<br>
<br>
The biggest challenges we've faced are<br>
<br>
- Elasticity of cloud - the perception from Amazon is that you can scale up and down. Within a standard private cloud on-premise, the elasticity can only come from removing other work (since we aim to be using the resources to the full). We use short queue,
opportunistic batch work to fill in so that we drain and re-instantiate the high throughput computing batch workers to accommodate some elasticity but it is limited. Spot market functionality would be interesting but we've not seen something in OpenStack yet.<br>
<br>
- Scheduling - We have more work to do that there are resources. The cloud itself has no 'queue' or fair share. The experiment workflows thus have to place their workload into the cloud within their quota and maintain a queue or work on their side. INFN are
working on some enhancements to OpenStack with Blazar or Nova queues which is worth following.<br>
<br>
- Quota - Given the fixed resources, the quota controls are vital to avoid overcommitting. Currently, quotas are flat so the cloud administrators are asked to adjust the quotas to balance the user priorities within their overall pledges. The developments in
Nested Projects coming along with Kilo will be a major help here and we're working with BARC to deliver this in Nova so an experiment resource co-ordinator can be given the power to manage quotas for their subgroups.<br>
<br>
- VM Lifecycle - We have around 200 arrivals and departures a month. Without a credit card, it would be easy for their compute resources to remain running after they have left. We have an automated engine which ensures that VMs of departing staff are quiesced
and deleted following a standard lifecycle.<br>
<br>
- Accounting - The cloud accounting systems are based on vCPUs. There is no concept in OpenStack of a relative performance measure so you could be allocated a VM on 3 year old hardware or the latest and the vCPU metric is the same. In the high throughput use
cases, there should be a relative unit which is used to scale with regards to the amount of work that could be done. With over 20 different hardware configurations running (competitive public procurement cycles over 4 years), we can't define 100 flavors with
different accounting rates and expect our users to guess which ones have capacity.<br>
<br>
We're starting to have a look at bare metal and containers too. Using OpenStack as an overall resource allocation system to ensure all compute usage is accounted and managed is of great interest. The highlighted items above will remain but hopefully we are
work with others in the community to address them (as we are with nested projects)<br>
<span style="color:#888888"><br>
<span class="hoenzb">Tim</span></span><o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><br>
<br>
<br>
<br>
> -----Original Message-----<br>
> From: Jonathon A Anderson [mailto:<a href="mailto:Jonathon.Anderson@Colorado.EDU">Jonathon.Anderson@Colorado.EDU</a>]<br>
> Sent: 13 January 2015 20:14<br>
> To: <a href="mailto:openstack-hpc@lists.openstack.org">openstack-hpc@lists.openstack.org</a><br>
> Subject: [openstack-hpc] How are you using OpenStack for your HPC workflow?<br>
> (We're looking at Ironic!)<br>
><br>
> Hi, everybody!<br>
><br>
> We at CU-Boulder Research Computing are looking at using OpenStack Ironic<br>
> for our HPC cluster(s). I’ve got a sysadmin background, so my initial goal is to<br>
> simply use OpenStack to deploy our traditional queueing system (Slurm), and<br>
> basically use OpenStack (w/Ironic) as a replacement for Cobbler, xcat, perceus,<br>
> hand-rolled PXE, or whatever else sysadmin-focused PXE-based node<br>
> provisioning solution one might use. That said, once we have OpenStack (and<br>
> have some competency with it), we’d presumably be well positioned to offer<br>
> actual cloud-y virtualization to our users, or even offer baremetal instances for<br>
> direct use for some of our (presumably most<br>
> trusted) users.<br>
><br>
> But how about the rest of you? Are you using OpenStack in your HPC workloads<br>
> today? Are you doing “traditional” virtualization? Are you using the nova-<br>
> compute baremetal driver? Are you using Ironic? Something even more exotic?<br>
><br>
> And, most importantly, how has it worked for you?<br>
><br>
> I’ve finally got some nodes to start experimenting with, so I hope to have some<br>
> real hands-on experience with Ironic soon. (So far, I’ve only been able to do<br>
> traditional virtualization on some workstations.)<br>
><br>
> ~jonathon<br>
><br>
> _______________________________________________<br>
> OpenStack-HPC mailing list<br>
> <a href="mailto:OpenStack-HPC@lists.openstack.org">OpenStack-HPC@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc</a><br>
_______________________________________________<br>
OpenStack-HPC mailing list<br>
<a href="mailto:OpenStack-HPC@lists.openstack.org">OpenStack-HPC@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>