Open Stack

On 09/21/2015 02:56 AM, Daniel P. Berrange wrote:
> On Fri, Sep 18, 2015 at 05:47:31PM +0000, Carlton, Paul (Cloud Services) wrote:
>> However the most significant impediment we encountered was customer
>> complaints about performance of instances during migration.  We did a little
>> bit of work to identify the cause of this and concluded that the main issues
>> was disk i/o contention.  I wonder if this is something you or others have
>> encountered?  I'd be interested in any idea for managing the rate of the
>> migration processing to prevent it from adversely impacting the customer
>> application performance.  I appreciate that if we throttle the migration
>> processing it will take longer and may not be able to keep up with the rate
>> of disk/memory change in the instance.
>
> I would not expect live migration to have an impact on disk I/O, unless
> your storage is network based and using the same network as the migration
> data. While migration is taking place you'll see a small impact on the
> guest compute performance, due to page table dirty bitmap tracking, but
> that shouldn't appear directly as disk I/O problem. There is no throttling
> of guest I/O at all during migration.

Technically if you're doing a lot of disk I/O couldn't you end up with a case 
where you're thrashing the page cache enough to interfere with migration?  So 
it's actually memory change that is the problem, but it might not be memory that 
the application is modifying directly but rather memory allocated by the kernel.

>> Could you point me at somewhere I can get details of the tuneable setting
>> relating to cutover down time please?  I'm assuming that at these are
>> libvirt/qemu settings?  I'd like to play with them in our test environment
>> to see if we can simulate busy instances and determine what works.  I'd also
>> be happy to do some work to expose these in nova so the cloud operator can
>> tweak if necessary?
>
> It is already exposed as 'live_migration_downtime' along with
> live_migration_downtime_steps, and live_migration_downtime_delay.
> Again, it shouldn't have any impact on guest performance while
> live migration is taking place. It only comes into effect when
> checking whether the guest is ready to switch to the new host.

Has anyone given thought to exposing some of these new parameters to the 
end-user?  I could see a scenario where an image might want to specify the 
acceptable downtime over migration.  (On the other hand that might be tricky 
from the operator perspective.)

Chris