Open Stack

> We had to do the same downgrade with openvswitch, the newest version, under heavy load, corrupts packets in-transit, but we do not have the time to investigate the issue further.

Daniele, what was the openvswitch version before and after the upgrade?  And which ethernet drivers do you have?  The corruption maybe related to the drivers you have (the issues may be triggered by the way openvswitch flows are configured in Icehouse vs Havana).

Thanks.

From: Daniele Venzano <daniele.venzano at eurecom.fr<mailto:daniele.venzano at eurecom.fr>>
Organization: Eurecom
Date: Sunday, October 19, 2014 11:46 PM
To: "openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>" <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: Re: [Openstack-operators] qemu 1.x to 2.0

We have the same setup (Icehouse on Ubuntu 12.04) and had similar issues. We downgraded qemu from 2.x to 1.x, as we cannot terminate all VMs for all users. We had non-resumable VMs also in the middle of the 1.x series and nothing was documented in the changlelog.
We had to do the same downgrade with openvswitch, the newest version, under heavy load, corrupts packets in-transit, but we do not have the time to investigate the issue further.

We plan to warn our users in time for the next major upgrade to Juno that all VMs need to be terminated, probably during the Christmas holidays. I do not think they will be happy.
Seeing also all the problems we had upgrading Neutron from OVS to ML2, terminating all VMs is probably the best policy anyway during an OpenStack upgrade. Or you do lots of migrations and upgrade qemu one compute host at the time, but if something goes wrong you end-up with an angry user and a stuck VM.

It certainly is a big deal.

On 10/20/14 00:59, Joe Topjian wrote:
Hello,

We recently upgraded an OpenStack Grizzly environment to Icehouse (doing a quick stop-over at Havana). This environment is still running Ubuntu 12.04.

The Ubuntu 14.04 release notes<https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#Ubuntu_Server> make mention of incompatibilities with 12.04 and moving to 14.04 and qemu 2.0. I didn't think that this would apply for upgrades staying on 12.04, but it indeed does.

We found that existing instances could not be live migrated (as per the release notes). Additionally, instances that were hard-rebooted and had the libvirt xml file rebuilt could no longer start, either.

The exact error message we saw was:

"Length mismatch: vga.vram: 1000000 in != 800000"

I found a few bugs that are related to this, but I don't think they're fully relevant to the issue I ran into:

https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1308756
https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321
https://bugs.launchpad.net/nova/+bug/1312133

We ended up downgrading to the stock Ubuntu 12.04 qemu 1.0 packages and everything is working nicely.

I'm wondering if anyone else has run into this issue and how they dealt with it or plan to deal with it.

Also, I'm curious as to why exactly qemu 1.x to 2.0 are incompatible with each other. Is this just an Ubuntu issue? Or is this native of qemu?

Unless I'm missing something, this seems like a big deal. If we continue to use Ubuntu's OpenStack packages, we're basically stuck at 12.04 and Icehouse unless we have all users snapshot their instance and re-launch in a new cloud.

Thanks,
Joe




_______________________________________________
OpenStack-operators mailing list
OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20141020/bae9190a/attachment.html>