Open Stack

On Sat, May 09, 2015 at 04:55:47PM +0000, Adrian Otto wrote:
 
> I will also mention that it’s natural to be allergic to the idea of
> nested virtualization. We all know that creating multiple levels of
> hardware virtualization leads to bad performance outcomes.

This seems to paint an overly bleak picture of nested virtualization.
Please allow me to explain.

Speaking from frequent upstream testing, KVM on KVM nested
virtualization, is far more performant[1][2] than nesting with plain
emulation (maybe you're referring to this). also stated by James
Bottomley elsewhere in this thread).

A small test (was done in OCT, 2013): 

    A _single_ `make` process (10 iterations) to compile Linux Kernel,
    with nested EPT[1] (Extended Page Tables) enabled. Perform the test
    on a baremetal host (L0); a guest hypervisor (L1), running on L0;
    and a nested guest (L2), running on L1.

Test script:

    ------------------------------------------------------
    #!/bin/bash
    cd /home/test/src/linux
    
    make defconfig
    
    for i in {1..10}; do
         make clean;
         sync;
         sudo sh -c "echo 3 > /proc/sys/vm/drop_caches"
         time make;
    done 2>&1 | tee make-timings.txt
    ------------------------------------------------------

Result of time to `make defconfig`, average of 10 iterations on:
    
    L2: 4m53s
    L1: 3m56s
    L0: 2m14s

As you can see from the timings, for some kind of workloads (e.g. CPU
intensive) KVM nested virt is not as bad.  And, it's worth noting that
this test was done without any other performance tuning. 

Of course, this doesn't mean that it's suitable for all use cases or
there's no complexity. This needs more testing and providing feedback to
upstream KVM/Kernel community. FWIW, in my experience of reporting bugs
in this area, upstream KVM maintainers have been very responsive.


[1] Talk on the said subject by Gleb Natapov, KVM maintainer in 2013 --
    http://www.linux-kvm.org/images/8/8c/Kvm-forum-2013-nested-ept.pdf
[2] Talk by Jan Kizka, Bandan Das et al. at 2014 KVM Forum, with details on
    performance evaluation --
    http://www.linux-kvm.org/images/3/33/02x03-NestedVirtualization.pdf

-- 
/kashyap

> However, "nested containers" do not carry that same drawback, because
> the actual overhead of a Linux cgroup and Kernel Namespeaces are much
> lighter than a hardware virtualization. There are cases where having a
> container-in-container setup gives compelling benefits. That’s why
> I’ll argue vigorously for both Nova and Magnum to be able to produce
> container instances both at the machine level, and allow Magnum to
> produce "nested containers” to produce better workload consolidation
> density. in a setup with no hypervisors at all.
> 
> To sum up, I strongly support merging in nova-docker, with the caveat
> that it operates within the existing Nova API (with few minor
> exceptions). For features that require API features that are truly
> container specific, we should land those in Magnum, and keep the Nova
> API scoped to operations that are appropriate for “all" instance
> types.