Open Stack

On Tue, Apr 05, 2016 at 02:27:30PM +0200, Roman Dobosz wrote:
> Hey all,
> 
> On yesterday's scheduler meeting I was raised the idea of bringing up 
> the FPGA to the OpenStack as the resource, which than might be exposed 
> to the VMs.
> 
> The use cases for motivations, why one want do this, are pretty broad -
> having such chip ready on the computes might be beneficial either for
> consumers of the technology and data center administrators. The
> utilization of the hardware is very broad - the only limitations are
> human imagination and hardware capability - since it might be used for
> accelerating execution of algorithms from compression and cryptography,
> through pattern recognition, transcoding to voice/video analysis and
> processing and all the others in between. Using FPGA to perform data
> processing may significantly reduce CPU utilization, the time and power
> consumption, which is a benefit on its own.
> 
> On OpenStack side, unlike utilizing the CPU or memory, for actually 
> using specific algorithm with FPGAs, it has to be programmed first. So 
> in simplified scenario, it might go like this:
> 
> * User selects VM with image which supports acceleration,
> * Scheduler selects the appropriate compute host with FPGA available,
> * Compute gets request, program IP into FPGA and then boot up the 
>   VM with accelerator attached.
> * If VM is removed, it may optionally erase the FPGA.
> 
> As you can see, it seems not complicated at this point, however it 
> become more complex due to following things we also have to take into 
> consideration:
> 
> * recent FPGA are divided into regions (or slots) and every of them 
>   can be programmed separately
> * slots may or may not fit the same bitstream (the program which FPGA
>   is fed, the IP)
> * There is several products around (Altera, Xilinx, others), which make 
>   bitstream incompatible. Even between the products of the same company
> * libraries which abstract the hardware layer like AAL[1] and their 
>   versions
> * for some products, there is a need for tracking memory usage, which 
>   is located on PCI boards
> * some of the FPGAs can be exposed using SR-IOV, while some other not, 
>   which implies multiple usage abilities

Along similar lines we have proposals to add vGPU support to Nova,
where the vGPUs may or may not be exposed using SR-IOV. We also want
to be able to on the fly decide whether any physical GPU is assigned
entirely to a guest as a full PCI device, or whether we only assign
individual "virtual functions" of the GPU. This means that even if
the GPU in question does *not* use SR-IOV, we still need to track
the GPU and vGPUs in the same way as we track PCI devices, so that
we can avoid assigning a vGPU to the guest, if the underlying physical
PCI device is already assigned to the guest.

I can see we will have much the same issue with FPGAs, where we may
either want to assign the entire physical PCI device to a guest, or
just assign a particular slot in the FPGA to the guest. So even if
the FPGA is not using SR-IOV, we need to tie this all into the PCI
device tracking code, as we are intending for vGPUs.

All in all, I think we probably ought to generalize the PCI device
assignment modelling so that we're actually modelling generic
hardware devices which may or may not be PCI based, so that we can
accurately track the relationships between the devices.

With NIC devices we're also seeing a need to expose capabilities
against the PCI devices, so that the schedular can be more selective
in deciding which particular devices to assign. eg so we can distinguish
between NICs which support RDMA and those which don't, or identify NIC
with hardware offload features, and so on. I can see this need to
associate capabilities with devices is something that will likely
be needed for the FPGA scenario, and vGPUs too. So again this points
towards more general purpose modelling of assignable hardware devices
beyond the limited PCI device modelling we've got today.

Looking to the future I think we'll see more usecases for device
assignment appearing for other types of device.

IOW, I think it would be a mistake to model FPGAs as a distinct
object type on their own. Generalization of assignable devices
is the way to go

> In other words, it may be necessary to incorporate another actions:
> 
> * properly discover FPGA and its capabilities
> * schedule right bitstream with corresponding matching unoccupied FPGA
>   device/slot
> * actual program FPGA
> * provide libraries into VM, which are necessary for interacting between
>   user program and the exposed FPGA (or AAL) (this may be optional, 
>   since user can upload complete image with everything in place)
> * bitstream images have to be keep in some kind of service (Glance?) 
>   with some kind of way for identifying which image match what FPGA
> 
> All of that makes modelling resource extremely complicated, contrary to 
> CPU resource for example. I'd like to discuss how the goal of having 
> reprogrammable accelerators in OpenStack can be achieved. Ideally I'd 
> like to fit it into Jay and Chris work on resource-*.

I think you shouldn't look at the FPGAs as being like CPU resource, but
rather look at them as a generalization of PCI device asignment.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|