<div dir="ltr">I think that workload management is perhaps too vague of a term here. There are a few different processes in play. <div><br></div><div>One is the system level resource management (nova-scheduler, etc). This component needs to figure out how to allocate the resources to tenants, and pull them back with needed. </div>
<div><br></div><div>Inside of a single tenant's allocation, there is a need for workload management. This is a good place for traditional HPC resource managers; we've run torque in this capacity, for example. </div>
<div><br></div><div>I think that the model today for nova scheduler is wrong; it only supports open-ended leases on compute resources. Another component that makes this tricky is the need to support interactive workloads; batch is good for well, batch, but interactive is always problematic in these environments. </div>
<div><br></div><div>There is some research targeting this problem, in particular Mesos and Omega. Mesos looks ok for serial workloads, but has a real problem with parallel (non-resizable) ones. Omega looks a lot better, but that is an internal google thing. I don't think that there is an off the shelf solution today. </div>
<div>-nld</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Jul 26, 2013 at 1:48 AM, Di Pe <span dir="ltr"><<a href="mailto:dipeit@gmail.com" target="_blank">dipeit@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">All,<div><br></div><div>one issue Joshua touched on was options for workload management. While IB and GPU seem to frequently discussed when it comes to openstack and HPC they are not so relevant in our HPC environment (3000 ish cores, each node connected via 1G, scaleout NAS storage, biomedical research, genomics, proteomics, statistics) ..... many other midsize shops may be have similar setups. We are just starting to look at openstack for a for a potential deployment with Ubuntu 14.04. We have good experiences using KVM for some of our resources. Some of the things we are hoping to get from openstack in the future are:</div>
<div><br></div><div><div><ul><li>flexible partitioning of resources for special sauce software (hadoop, interactive HPC software)</li><li>self service for developers and scientists</li><li>allow research group that spans multiple research organizations (internal / external) controlled access to an isolated (virtual) datacenter (potentially with fisma compliance)</li>
<li>save images that researchers built for later use (reproducible in case someone asks how they got to this result)</li><li>chargeback for HPC resources for internal and external users</li><li>usage of idle resources for testing in Enterprise IT, VDI ,etc</li>
<li>compute fencing (as we are heading to 24 cores per box most of our multi threaded code can still only take advantage of 4-6 cores. This either leaves stuff idle or users step on each other on shared nodes ....cgroups is a bit of a pain to maintain.</li>
<li>checkpointing and restarting long running jobs (for prioritization and better protection against job failures). perhaps LXC containers as alternative to KVM (we use BLCR today but that community is quite small)</li><li>
standardization of our infrastructure</li><li>potential participation in futuregrid, XSEDE, etc</li></ul></div><br></div><div>That's perhaps a lot to ask but we would be looking at a 2-3 year time frame. What I don't quite unterstand, how would one handle workload management. Currently we see people using SGE, moab, LSF and some slurm in classic HPC. Concepts like backfill, preemption fair share and such things are probably unknown to openstack? If so it would perhaps be acceptable to run a workload manager on a subset of always-on KVM systems or even baremetal for classic HPC. But how does one consolidate the reporting, billing, chargeback of 2 separate systems? </div>
<div><br></div><div>Are there any efforts to integrate workload managers directly into nova? SGE and Slurm are both open source and would support everything we require. Or are folks thinking about writing something from scratch in python?</div>
<div><br></div><div>Thanks</div><div>dipe</div><div><br></div><div><br></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Jul 25, 2013 at 6:22 PM, Narayan Desai <span dir="ltr"><<a href="mailto:narayan.desai@gmail.com" target="_blank">narayan.desai@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Brian's right. You will end up doing a lot of work; nova isn't ready for this out of the box. <div>
<br></div><div>The key problems are:</div><div> - drivers for virtualization (either via SR-IOV or device passthrough) for net + gpu</div>
<div> - IO architecture</div><div><br></div><div>There is apparently SR-IOV support for IB, provided that you have the right hardware, firmware, and driver, though I haven't managed to make it work. This provides a pkey isolated multi-tenant environment. THat is basically a mellanox only solution. </div>
<div><br></div><div>Like Brian said, Xen is the only way to go for GPU virtualization. </div><div><br></div><div>You can do some really interesting things with the I/O architecture. We've been experimenting with both glusterfs and ceph. Both seem to perform decently well; we've managed to get the glusterfs setup going at 60 GB/s in aggregate with a pile of clients. There isn't good integration of all of the capabilities yet in mainline openstack, but this looks promising. Ceph looks like the mainline integration is better, but we haven't tried those things out yet.</div>
<div><br></div><div>At the end of the day, you need to ask yourself why you want to accomplish this. If you're running a workload that is well suited to an HPC cluster, you should probably use one. If you need multi-tenancy, user control of system software, or need to run a workload poorly suited for a traditional cluster, then it is worth thinking strongly about. You'll end up needing to do a bunch of work though.</div>
<div><br></div><div>In our case, the reason that we pursued this course is because we have workloads and developers that benefit from the cloud control plane.</div><div><br></div><div>hth.</div><div> -nld</div><div><br></div>
</div><div class="gmail_extra"><br><br><div class="gmail_quote"><div>On Thu, Jul 25, 2013 at 3:30 PM, Brian Schott <span dir="ltr"><<a href="mailto:brian.schott@nimbisservices.com" target="_blank">brian.schott@nimbisservices.com</a>></span> wrote:<br>
</div><div><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div>Joshua,</div><div><br></div><div>It is something those of us working the gap between HPC and cloud have struggled with. We lack a strong project team within OpenStack IMHO, although there are several small groups doing HPC with OpenStack. Hopefully others will chime in on some other topics, such as Infiniband support, but we did some testing with a GRID K2 card for GPU pass-through with different hypervisors. A talk I gave at the OpenStack DC meetup is here:</div>
<div><a href="http://www.slideshare.net/bfschott/nimbis-schott-openstackgpustatus20130618" target="_blank">http://www.slideshare.net/bfschott/nimbis-schott-openstackgpustatus20130618</a></div><div><br></div><div>The short GPU answer is that it is possible with Xen, XenCenter, and XCP to passthrough GPUs today, but OpenStack doesn't have support by default in Nova. This is still in a roll-your-own mode for deployment.</div>
<div><br></div><div>Brian</div><div><span style="font-size:12px"><br></span></div><div><span style="font-size:12px">-------------------------------------------------</span></div><div><span style="border-collapse:separate;border-spacing:0px"><div>
<div style="font-size:12px">Brian Schott, CTO</div><div style="font-size:12px">Nimbis Services, Inc.</div><div style="font-size:12px"><a href="mailto:brian.schott@nimbisservices.com" target="_blank">brian.schott@nimbisservices.com</a></div>
<div style="font-size:12px">ph: <a href="tel:443-274-6064" value="+14432746064" target="_blank">443-274-6064</a> fx: <a href="tel:443-274-6060" value="+14432746060" target="_blank">443-274-6060</a></div></div><div><br></div>
</span><br>
</div>
<br><div><div><div><div>On Jul 25, 2013, at 4:06 PM, Joshua Dotson <<a href="mailto:josh@knoesis.org" target="_blank">josh@knoesis.org</a>> wrote:</div><br></div></div><blockquote type="cite"><div><div>
<div dir="ltr">Hello.<br><br>A contingent of my organization, the <a href="http://www.knoesis.org/" target="_blank">Kno.e.sis Center @ Wright State University</a>, recently received a grant award which we intend to use to
support a handful of mid-size HPC-style workloads (MPI <--
definitely, GPGPU <-- if possible/plausible) in addition to many mid-size
IaaS-style workloads (MongoDB, Storm, Hadoop, many others). As a third layer, I'm playing with the idea of evaluating an elastic OpenShift Origin atop the same infrastructure. Approximately $400k to $500k will hopefully be available for this deployment, though
exact numbers are not yet available to me. <br>
<br>While I'm prepared to build a home-grown small-to-mid-size
"classical" HPC, using modern hardware, and a smaller silo for
home-grown Openstack for the minority stakeholders, I am hoping to find ways of making proponents of
both workloads simultaneously happy, or close to it. That is, I would like to give my
computer scientist users a friendly method of running their HPC-style
jobs on a combined performance-tuned silo of Openstack. Doing so would
load-balance the procured hardware and infrastructure with the users who
want a Tomcat or a Virtuoso instance.<br>
<br>I see a number of serious issues realizing such a
goal. For example, the state of Infiniband vs. Openstack seems not
quite ready/available/documented/accessible for such use in production, unless I'm just blind to the right blogs. The added myriad
abstractions and latency virtualization might impose on an HPC task, not to mention cloud
software defined networking (Quantum, especially when sans hardware acceleration), seem likely to really get in the way of practicality, economics and efficiency. That said, most of what we do here isn't HPC, so I
believe such trade-offs can be agreed upon, if a reasonable job
scheduling and workload management mechanism can be found and agreed upon by all stake holders, grant proposal majority (HPC) and minority (IaaS) alike. <br><br>I get the impression from my readings that HPC-style
deployment (separate from job queuing) against the EC2 API should work. I don't have a good feeling that the experience would be particularly
friendly, however, without paying for closed source applications. I'm thinking a high-performance Ceph install would help bring up the storage end of things in a modern open-source CoTS way. I've not done specific research on Lustre + Openstack, but no reports of such a setup have presented themselves to me, either.<br>
<br>These blue sky ideas matter nil, it seems, if a sufficiently-large high-performance
production-quality Openstack deployment is beyond the funds
to be allotted, which is something else I'm working on. I've built smallish but useful virt-manager, oVirt and
Openstack environments here already, but none of them are enough for the very-important HPC job proposed for this grant. The
scientist running the proposed computation gave me following information to
clarify what would parity (for his job only) his experience running the
computation with an external HPC service provider.<br>
<ul><li>MPI</li><li>20 Gbps Infiniband compute interconnect<br></li><li>600 cores (those currently used are probably G4 Opteron 2.4 Ghz)<br></li><li>4 GB RAM per core</li><li>at lest 2 TB shared storage, though I'm thinking we need much more for use by our general community</li>
<li>unsure of the storage networking topology<br></li></ul><p>We're in the shopping phase for this
grant award and are still playing with ideas. It seems likely to snap back into an old-school HPC, at this time. I've fielded communication about our needs to a number of Openstack and hardware providers, on the hope that they can bring something helpful to the table.<br>
</p><div>Please let know if you can point me in the right direction(s). I'm up to reading whatever text is thrown at me on this topic. :-)<br><br></div>Thanks,<br>Joshua<br><div>-- <br>Joshua M. Dotson<br>Systems Administrator<br>
Kno.e.sis Center<br>Wright State University - Dayton, OH<br></div></div></div></div>
_______________________________________________<br>OpenStack-HPC mailing list<br><a href="mailto:OpenStack-HPC@lists.openstack.org" target="_blank">OpenStack-HPC@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc</a><br>
</blockquote></div><br></div><br>_______________________________________________<br>
OpenStack-HPC mailing list<br>
<a href="mailto:OpenStack-HPC@lists.openstack.org" target="_blank">OpenStack-HPC@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc</a><br>
<br></blockquote></div></div></div><br></div>
<br>_______________________________________________<br>
OpenStack-HPC mailing list<br>
<a href="mailto:OpenStack-HPC@lists.openstack.org" target="_blank">OpenStack-HPC@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc</a><br>
<br></blockquote></div><br></div>
</div></div><br>_______________________________________________<br>
OpenStack-HPC mailing list<br>
<a href="mailto:OpenStack-HPC@lists.openstack.org">OpenStack-HPC@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc</a><br>
<br></blockquote></div><br></div>