Open Stack

On 2017/07/27 20:53, Mooney, Sean K wrote:
> 
> 
>> -----Original Message-----
>> From: TETSURO NAKAMURA [mailto:nakamura.tetsuro at lab.ntt.co.jp]
>> Sent: Thursday, July 27, 2017 2:13 AM
>> To: Daniel P. Berrange <berrange at redhat.com>; Mooney, Sean K
>> <sean.k.mooney at intel.com>
>> Cc: Jay Pipes <jaypipes at gmail.com>; OpenStack Development Mailing List
>> (not for usage questions) <openstack-dev at lists.openstack.org>;
>> sfinucan at redhat.com; mriedemos at gmail.com; 【社内】【ML】srv-apl-arch <srv-
>> apl-arch at lab.ntt.co.jp>
>> Subject: Re: [srv-apl-arch:7353] Re: [openstack-dev] [nova] Discussions
>> for ivshmem support in OpenStack Nova
>>
>> On 2017/07/27 0:58, Daniel P. Berrange wrote:
>>> On Wed, Jul 26, 2017 at 11:53:06AM -0400, Jay Pipes wrote:
>>>> On 07/26/2017 09:57 AM, Daniel P. Berrange wrote:
>>>>> On Wed, Jul 26, 2017 at 09:50:23AM -0400, Jay Pipes wrote:
>>>>>> On 07/26/2017 03:06 AM, TETSURO NAKAMURA wrote:
>>>>>>> Hi Nova team,
>>>>>>>
>>>>>>> It has been quite a long time since the last discussion, but let
>>>>>>> me make sure one thing about the thread below.
>>>>>>>
>>>>>>> IIUC, Nova is not welcome ivshmem support because it is no longer
>>>>>>> supported by DPDK+QEMU.
>>>>>>>
>>>>>>> But how would you say if it is supported out of DPDK-tree and can
>>>>>>> be used from the newest qemu version ?
>>>>>>>
>>>>>>> We are now developing SPP, a DPDK-based vswitch, and thinking
>>>>>>> about trying to implement ivshmem support under our SPP code tree
>>>>>>> if nova (or at first libvirt community) is acceptable for ivshmem
>> configuration.
>>>>>>>
>>>>>>> Your advice will be very helpful for our decision-making in our
>> project.
>>>>>>
>>>>>> I think this is a question that the libvirt community would first
>>>>>> need to weigh in on since Nova is downstream from libvirt -- at
>>>>>> least in the sense of low-level hypervisor support.
>>>>>
>>>>> Libvirt already supports ivshmem device config
>>>>>
>>>>>      http://libvirt.org/formatdomain.html#elementsShmem
>>>>
>>>> Sorry, I suppose I should have said QEMU, not libvirt. Daniel, you
>>>> were the one that specifically discouraged doing anything on ivshmem
>> to Tetsuro:
>>>>
>>>> http://lists.openstack.org/pipermail/openstack-dev/2017-
>> July/120136.h
>>>> tml
>>>
>>> 'ivshmem' was the original device in QEMU and that is indeed still
>>> deprecated.
>>>
>>> There are now two replacements 'ivshmem-plain' and 'ivshmem-doorbell'
>>> which can be used instead, which are considered supported by QEMU,
>>> though most people will still recommend using 'vhostuser' instead if
>>> the use of ivshmem is at all network related.
>>>
>>> Regards,
>>> Daniel
>>>
>>
>> Thank you very much for the information about current status of ivshmem
>> in QEMU.
>> I now understand that 'ivshmem', 'ivshmem-plain' and 'ivshmem-doorbell'
>> are different solutions, and libvirt already supports the latter two.
>>
>> + Mr. Sean Mooney
>> Did you mean that you caution against building new solutions ontop of
>> 'ivshmem' or ontop of 'ivshmem-plain' and 'ivshmem-doorbell' too?
> [Mooney, Sean K] I would caution against building any networking based uscases ontop of ivshmem.
> The move to using a memdev instead of directly specifying shm args to qemu for ivshmem-plain/doorbell
> Should allow hugepage memory to be used instead of posix sheared mem which would be needed for spp.
> 
> That said just because you can use ivshmem-plain/doorbell with hugepages via a memdev form the qemu commandline
> does not mean it is supported by Libvirt https://libvirt.org/formatdomain.html#elementsShmem or that it is the best approach,
> upstream development has shifted to virtio and vhost-user.
> 0 copy tx from a vm is already supported with vhost-user and ovs-dpdk.
> 0 copy rx Is under development which is the main delta in performance between ivshmem and vhost-user that remains.
> Both of these features could be ported to the spp.
> 
> Vhost-user does have some inherent overhead such as the descriptor rings etc required to support virtio
> Must be created which is required to support the virtio spec which give you portability and performance when
> Coupled with multi queue and the dpdk vhost pmd in the guest.
> 
> If vhost-user is really not sufficient for your usecase I would first suggest extending Libvirt to allow
> Passing a memdev name as a parameter to the shmem element.
> At that point we could discuss how to request openstack to use those new element to create the shmem device.
> 
> The best approach using what openstack support today for a generic shmem dev would be via a Libvirt option in the image metatdata the same
> Way that serial ports allocation or vGPU memory is configured. If this device was to be used as a nic however
> A different approach of creating a new vnic_type of ivsheme_plain and ivshmem_doorbell would be more approiate with the
> Port binding profile used to transport the memory mapping info similar to how we expose info for sriov devices.
> 
> The issue with ivshsmem interfaces in a neutron context is its just a pci device where bar2 points to a shared memory region
> There is no way to pass info such as a mac address to a hypervisor for it to enforce so the use of ivshmem deveices as
> Nic means that you are enabling a adressless port that can transmit and recive any  packet unless you explicitly implement
> Port security (mac/arp anti spoof) in the vswitch backend in this case spp. That is why ivshmem was originally
> Rejected in icehouse by nova.
> 
>>
>> --
>> Tetsuro Nakamura <nakamura.tetsuro at lab.ntt.co.jp> NTT Network Service
>> Systems Laboratories
>> TEL:0422 59 6914(National)/+81 422 59 6914(International) 3-9-11,
>> Midori-Cho Musashino-Shi, Tokyo 180-8585 Japan
>>
> 

Hi Sean,

Thank you very much for information.
Now it is clear that if we start working on ivshmem, the first target is 
Libvirt to support memdev.
But it seems that before starting we need to check out the zero-copy 
feature of vhost-user.

 > 0 copy tx from a vm is already supported with vhost-user and ovs-dpdk.
 > 0 copy rx Is under development which is the main delta in performance 
between ivshmem and vhost-user that remains.
 > Both of these features could be ported to the spp.

Could you inform me if there is a document or a reference where I can 
get further info to enable these features and when these features are 
merged?
We have only tested ovs-dpdk2.5 with dpdk16.04.

-- 
Tetsuro Nakamura <nakamura.tetsuro at lab.ntt.co.jp>
NTT Network Service Systems Laboratories
TEL:0422 59 6914(National)/+81 422 59 6914(International)
3-9-11, Midori-Cho Musashino-Shi, Tokyo 180-8585 Japan