<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">The version we are using is:<br>
1.10.2-0ubuntu2~cloud0<br>
<br>
The version that was not working for us is:<br>
2.0.1+git20140120-0ubuntu2~cloud1<br>
<br>
Network:<br>
Intel Corporation I350 Gigabit Network Connection (igb module)<br>
<br>
We were seeing the problem, strangely enough, at the application
level, inside the VMs, where Hadoop was reporting corrupted data
on TCP connections. No other messages on the hypervisor or in the
VM kernel. Hadoop makes lots of connections to lots of different
VMs moving lots (terabytes) of data as fast as posssibile. Also,
it was non-deterministic, Hadoop would try several times to
transfer the data, sometimes successfully, sometimes giving up. I
tried some quick iperf tests, but they worked fine.<br>
<br>
Daniele<br>
<br>
On 10/20/14 18:46, Manish Godara wrote:<br>
</div>
<blockquote cite="mid:D06A89CE.44265%25manishg@yahoo-inc.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<div>> We had to do the same downgrade with openvswitch, the
newest version, under heavy load, corrupts packets in-transit,
but we do not have the time to investigate the issue further.</div>
<div><br>
</div>
<div>Daniele, what was the openvswitch version before and after
the upgrade? And which ethernet drivers do you have? The
corruption maybe related to the drivers you have (the issues may
be triggered by the way openvswitch flows are configured in
Icehouse vs Havana).</div>
<div><br>
</div>
<div>Thanks.</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt;
text-align:left; color:black; BORDER-BOTTOM: medium none;
BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT:
0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid;
BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Daniele Venzano
<<a moz-do-not-send="true"
href="mailto:daniele.venzano@eurecom.fr">daniele.venzano@eurecom.fr</a>><br>
<span style="font-weight:bold">Organization: </span>Eurecom<br>
<span style="font-weight:bold">Date: </span>Sunday, October
19, 2014 11:46 PM<br>
<span style="font-weight:bold">To: </span>"<a
moz-do-not-send="true"
href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>"
<<a moz-do-not-send="true"
href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re:
[Openstack-operators] qemu 1.x to 2.0<br>
</div>
<div><br>
</div>
<div>
<div bgcolor="#FFFFFF" text="#000000">
<div class="moz-cite-prefix">We have the same setup
(Icehouse on Ubuntu 12.04) and had similar issues. We
downgraded qemu from 2.x to 1.x, as we cannot terminate
all VMs for all users. We had non-resumable VMs also in
the middle of the 1.x series and nothing was documented in
the changlelog.<br>
We had to do the same downgrade with openvswitch, the
newest version, under heavy load, corrupts packets
in-transit, but we do not have the time to investigate the
issue further.<br>
<br>
We plan to warn our users in time for the next major
upgrade to Juno that all VMs need to be terminated,
probably during the Christmas holidays. I do not think
they will be happy.<br>
Seeing also all the problems we had upgrading Neutron from
OVS to ML2, terminating all VMs is probably the best
policy anyway during an OpenStack upgrade. Or you do lots
of migrations and upgrade qemu one compute host at the
time, but if something goes wrong you end-up with an angry
user and a stuck VM.<br>
<br>
It certainly is a big deal.<br>
<br>
On 10/20/14 00:59, Joe Topjian wrote:<br>
</div>
<blockquote
cite="mid:CA+y7hvhC0FnDR2M=tzYJchMQbLPLac78EwwhJLw0kS2KfihhBA@mail.gmail.com"
type="cite">
<div dir="ltr">Hello,
<div><br>
</div>
<div>We recently upgraded an OpenStack Grizzly
environment to Icehouse (doing a quick stop-over at
Havana). This environment is still running Ubuntu
12.04.</div>
<div><br>
</div>
<div>The <a moz-do-not-send="true"
href="https://wiki.ubuntu.com/TrustyTahr/ReleaseNotes#Ubuntu_Server">
Ubuntu 14.04 release notes</a> make mention of
incompatibilities with 12.04 and moving to 14.04 and
qemu 2.0. I didn't think that this would apply for
upgrades staying on 12.04, but it indeed does.</div>
<div><br>
</div>
<div>We found that existing instances could not be live
migrated (as per the release notes). Additionally,
instances that were hard-rebooted and had the libvirt
xml file rebuilt could no longer start, either.</div>
<div><br>
</div>
<div>The exact error message we saw was:</div>
<div><br>
</div>
<div>"Length mismatch: vga.vram: 1000000 in != 800000"</div>
<div><br>
</div>
<div>I found a few bugs that are related to this, but I
don't think they're fully relevant to the issue I ran
into:</div>
<div><br>
</div>
<div><a moz-do-not-send="true"
href="https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1308756">https://bugs.launchpad.net/ubuntu/+source/qemu/+bug/1308756</a><br>
</div>
<div><a moz-do-not-send="true"
href="https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321">https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1291321</a><br>
</div>
<div><a moz-do-not-send="true"
href="https://bugs.launchpad.net/nova/+bug/1312133">https://bugs.launchpad.net/nova/+bug/1312133</a><br>
</div>
<div><br>
</div>
<div>We ended up downgrading to the stock Ubuntu 12.04
qemu 1.0 packages and everything is working nicely.</div>
<div><br>
</div>
<div>I'm wondering if anyone else has run into this
issue and how they dealt with it or plan to deal with
it.</div>
<div><br>
</div>
<div>Also, I'm curious as to why exactly qemu 1.x to 2.0
are incompatible with each other. Is this just an
Ubuntu issue? Or is this native of qemu?</div>
<div><br>
</div>
<div>Unless I'm missing something, this seems like a big
deal. If we continue to use Ubuntu's OpenStack
packages, we're basically stuck at 12.04 and Icehouse
unless we have all users snapshot their instance and
re-launch in a new cloud.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Joe</div>
<div><br>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-operators mailing list
<a moz-do-not-send="true" class="moz-txt-link-abbreviated" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><a moz-do-not-send="true" class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a></pre>
</blockquote>
<br>
<br>
</div>
</div>
</span>
</blockquote>
<br>
<br>
</body>
</html>