- OpenStack-HPC - lists.openstack.org

Re: [openstack-hpc] Infiniband + Openstack?
by John Paul Walters 25 Jan '13

25 Jan '13

Álvaro, I had tried to use straight PCI passthrough with our hardware and didn't succeed in getting the stock OFED drivers to initialize properly. It could be that because our cards have the updated SR-IOV firmware on them, that it was breaking this functionality. But no, we don't yet have anything we can share. I hope that we have it soon, and I'll certainly let everyone know when we do. best, JP On Jan 25, 2013, at 10:04 AM, Álvaro López García <alvaro.lopez.garcia(a)cern.ch> wrote: > Hello there, John. > > On Wed 23 Jan 2013 (16:02), John Paul Walters wrote: >> Z, >> >> We're working on integrating IB into OpenStack via SR-IOV. It's not quite there yet, but we expect to have it working within a month or so. Right now, I'm unaware of anyone who has IB working inside of a VM, that's what we're developing. To do this, you need a few things: 1) a host that supports SR-IOV, 2) a ConnectX-2 or ConnectX-3 IB card, and 3) Alpha SR-IOV-enabled firmware for the cards as well as the corresponding SR-IOV-enabled OFED. Both of those (the firmware and OFED) need to come from Mellanox at the moment - they're not yet released. We've so far been successful in getting RDMA working inside of the VM; however, oddly, IPoIB doesn't work (the ports are recognized as being connected). We hope that this will get worked out in later firmware/OFED releases. > > We've IB inside our VMs since long, using direct passthrough of the device > into the VM without SR-IOV (so we only can run 1 VM per card). > >> Be aware that this isn't a Quantum integration, at least not yet. We're going to manage SR-IOV VIFs as resources and schedule accordingly. I know that there are folks using IB for image distribution and perhaps also for block storage. Our target is to get IB into the VM. > > In our group we're really interested on this. We had some work done in > the past about that (setting the card as a resource that could be attached > to the machines directly) but we had to abandon it because of lack of > time. Do you have something working? > > For us this worked independetly of SR-IOV, since in our PoC we only > specified the resources that could be used for each of the compute nodes > (i.e. the PCI addresses) and add them to a pool. Then the user could > request resources from this pool. > > Regards, > -- > Álvaro López García aloga(a)ifca.unican.es > Instituto de Física de Cantabria http://devel.ifca.es/~aloga/ > Ed. Juan Jordá, Campus UC tel: (+34) 942 200 969 > Avda. de los Castros s/n > 39005 Santander (SPAIN) > _____________________________________________________________________ > "I am always doing that which I cannot do, in order that I may learn > how to do it." -- Pablo Picasso

1 0

Infiniband + Openstack?
by zebra z zebra 24 Jan '13

24 Jan '13

Hi. I've been trying to get some simple/straight answers on IB for OpenStack for a few days now. It could well be that I've just been posting to the wrong list! ;) I'm considering a reasonably sized OpenStack deployment shortly (several thousand cores). One of the decisions I need to make centres around the type of interconnect I will use to connect the compute nodes, Swift objectstorage, shared storage, swift proxies et al. I have been heavily considering FDR infiniband for this task, but, from what I can see, IB has only recently become something one might consider stable for use in OpenStack. My peers are suggesting to me that I should just stick with 10GbE interconnects and be happy. In doing some background reading, it looks like somebody asked similar questions a year or two ago, and the general sentiment is that it'd work in IPoIB mode as just a very fast ethernet device, but it wouldn't really have any of the benefits of RDMA communication, or any such capability - and the issue at the time was that no python libraries had any understanding or concept of it. I'd like to hear the experiences of people who have or are using IB, or who have experiences with IB deployments. Thank you. --z

4 4

High bandwidth networking with openstack
by Narayan Desai 28 Nov '12

28 Nov '12

I've finally finished my writeup describing the experiments that we performed using Openstack to drive a wide area 100 gigabit network. I've included all of the details for configuration and tuning, as well as speculation why we're seeing such good numbers. tl;dr: you can push a whole lot of network bandwidth out of an openstack deployment with kvm/virtio; we hit 98 gigabits peak with 10 instances across 50 ms of latency. The post is here: http://buriedlede.blogspot.com/2012/11/driving-100-gigabit-network-with.html I'm happy to discuss this in more detail, if there are additional questions. -nld

1 0

Ceph + Nova
by JuanFra Rodríguez Cardoso 21 Nov '12

21 Nov '12

Hi everyone: I'd like to know your opinion as nova experts: Would you recommend CephFS as shared storage in /var/lib/nova/instances? Another option it would be use GlusterFS or MooseFS for /var/lib/nova/instances directory and Ceph RBD for Glance and Nova volumes, don't you think? Thanks for your attention. Best regards, JuanFra

1 0

Lunch today at the summit
by Tim Bell 18 Oct '12

18 Oct '12

I'll be meeting up with John Paul Waters from ISI at 12:40 today (Thursday). Others who are interested in an HPC/HTC chat over lunch are welcome to join us. We'll meet at the registration area on the 2nd floor and then we can go and find a queue. Tim

1 0

HPC session during upcoming Design Summit
by David Kang 15 Oct '12

15 Oct '12

Hi all, We will have a design summit session during next week's OpenStack Summit in San Diego. It'll at 11:00 am Tuesday, October 16. Its title is "Scheduler for HPC with OpenStack". We have asked to change the title to "HPC for OpenStack". We will cover wide range of HPC for OpenStack including * HPC extension current state * Accelerator support * Baremetal * Networking * IB * HPFS (e.g. Lustre) * Scheduler extensions * Community Requests/Open Discussion Please suggest other interesting topics and share your thoughts in the etherpad. (http://etherpad.openstack.org/GrizzlyHPC) Hope to see you there. Thanks, David ---------------------- Dr. Dong-In "David" Kang Computer Scientist USC/ISI

5 10

[HPC] monthly telecon
by John Paul Walters 28 Sep '12

28 Sep '12

Hi All, The monthly HPC telecon is postponed until we have a better sense of the sessions that will be scheduled for the Grizzly Design Summit. We'll shoot for holding the telecon on Monday, Oct. 8 in the hopes that we'll know whether our HPC sessions were accepted. If the decisions aren't yet made, we'll re-evaluate. The intent of this month's telecon will be to prepare for the Grizzly Summit. best, JP

1 0

Re: [openstack-hpc] [Openstack] [HPC] Monthly telecon
by Patrick Petit 11 Sep '12

11 Sep '12

Hi JP, Sorry, I won't be able to join on the 10th. I think it'll be useful to take an action to make sure the HPC extensions (i.e. extra specs extensions and capabilities filtering) pushed into Folsom are properly documented in the OpenStack Admin Guide. I think it's not the case right now especially around how to specify off specs capabilities (nova.conf versus not nova.conf ?). For example, it's been said that key values for things like capabilities can be defined via host aggregates but it's not described how... Discussions and info around these topics are scattered and also the documentation is incomplete and lacking practical details. Thanks Patrick 2012/9/3 John Paul Walters <jwalters(a)isi.edu> > Hi, > > The HPC telecon, normally scheduled for the first Monday of the month, > will instead be held on Monday, Sep. 10 due to the Labor Day holiday in the > US. It'll be held at 12:00 noon Eastern Time. I'll follow up with an > agenda near the end of next week. If there's anything that others would > like to place on the agenda, please let me know. > > Separately, I'd like to bring to your attention that Brian Schott has > succeeded in getting a dedicated HPC mailing list started. For anyone > interested, please sign up here: > > openstack-hpc(a)lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-hpc > > best, > JP > > _______________________________________________ > Mailing list: https://launchpad.net/~openstack > Post to : openstack(a)lists.launchpad.net > Unsubscribe : https://launchpad.net/~openstack > More help : https://help.launchpad.net/ListHelp > > -- *"Give me a place to stand, and I shall move the earth with a lever"*

2 2

[HPC] Reminder monthly telecon Sep. 10
by John Paul Walters 10 Sep '12

10 Sep '12

Hi, This is a reminder that we'll hold our next monthly HPC telecon this coming Monday, Sep. 10 and 12:00 noon Eastern Time. We'll use webex (details below). The agenda is somewhat open. Our default will be to start the conversation about HPC features that folks are interested in adding to the Grizzly release. If anyone has any other specific agenda items, they're welcome to propose them. I'm unable to attend, so my colleague David Kang will be hosting this meeting. We look forward to talking to you! best, JP John Paul Walters invites you to attend this online meeting. Topic: HPC Monthly Telecon Date: Monday, September 10, 2012 Time: 12:00 pm, Eastern Daylight Time (New York, GMT-04:00) Meeting Number: 927 246 497 Meeting Password: hpcmonthly ------------------------------------------------------- To join the online meeting (Now from mobile devices!) ------------------------------------------------------- 1. Go to https://openstack.webex.com/openstack/j.php?ED=203524102&UID=1431607857&PW=… 2. If requested, enter your name and email address. 3. If a password is required, enter the meeting password: hpcmonthly 4. Click "Join". To view in other time zones or languages, please click the link: https://openstack.webex.com/openstack/j.php?ED=203524102&UID=1431607857&PW=…

5 6

weex address for today's meeting
by Joseph Suh 10 Sep '12

10 Sep '12

All, Today's hpc meeting will be at https://usc-isi.webex.com/mw0306ld/mywebex/default.do?siteurl=usc-isi Password is dodcs. Thanks, Joseph ---- (w) 703-248-6160 (c) 571-340-2434 (f) 703-812-3712 http://www.east.isi.edu/~jsuh Information Sciences Institute University of Southern California 3811 N. Fairfax Drive Suite 200 Arlington, VA, 22203, USA

1 0