Hi.
I've been trying to get some simple/straight answers on IB for OpenStack for a few days now. It could well be that I've just been posting to the wrong list! ;)
I'm considering a reasonably sized OpenStack deployment shortly (several thousand cores). One of the decisions I need to make centres around the type of interconnect I will use to connect the compute nodes, Swift objectstorage, shared storage, swift proxies et al.
I have been heavily considering FDR infiniband for this task, but, from what I can see, IB has only recently become something one might consider stable for use in OpenStack. My peers are suggesting to me that I should just stick with 10GbE interconnects and be happy.
In doing some background reading, it looks like somebody asked similar questions a year or two ago, and the general sentiment is that it'd work in IPoIB mode as just a very fast ethernet device, but it wouldn't really have any of the benefits of RDMA communication, or any such capability - and the issue at the time was that no python libraries had any understanding or concept of it.
I'd like to hear the experiences of people who have or are using IB, or who have experiences with IB deployments.
Thank you.
--z