Open Stack

I've been trying to figure this one out for a while, so I'll try and be as thorough as possible in this post but apologies if I miss anything pertinent out.
 
First off, I'm running a set up with one control node and 5 compute nodes, all created using the Stackgeek scripts - http://www.stackgeek.com/guides/gettingstarted.html. The first two (compute1 and compute 2) were created at the same time, compute3, 4 and 5 were added as needed later. My VMs are predominantly CentOS, while my Openstack nodes are Ubuntu 14.04.1
 
The symptom: irregular high latency/packet loss to VMs on all compute boxes except compute3. Mostly a pain when trying to do anything via ssh on a VM because the lag makes it difficult to do anything, but it shows itself quite nicely through pings as well:
--- 10.0.102.47 ping statistics ---
111 packets transmitted, 103 received, 7% packet loss, time 110024ms
rtt min/avg/max/mdev = 0.096/367.220/5593.100/1146.920 ms, pipe 6
 

I have tested these pings:
VM to itself (via its external IP) seems fine
VM to another VM is not fine
Hosting compute node to VM is not fine
My PC to VM is not fine (however the other way round works fine)
 
 
Top on a (32 core) compute node with laggy VMs:
top - 12:09:20 up 33 days, 21:35,  1 user,  load average: 2.37, 4.95, 6.23
Tasks: 431 total,   2 running, 429 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.6 us,  3.4 sy,  0.0 ni, 96.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65928256 total, 44210348 used, 21717908 free,   341172 buffers
KiB Swap:  7812092 total,  1887864 used,  5924228 free.  7134740 cached Mem
 
And for comparison, on the one compute node that doesn't seem to be suffering from this:
top - 12:12:20 up 33 days, 21:38,  1 user,  load average: 0.28, 0.18, 0.15
Tasks: 399 total,   3 running, 396 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.1 sy,  0.0 ni, 98.9 id,  0.6 wa,  0.0 hi,  0.0 si,  0.0 st
KiB Mem:  65928256 total, 49986064 used, 15942192 free,   335788 buffers
KiB Swap:  7812092 total,   919392 used,  6892700 free. 39272312 cached Mem
 
Top on a laggy VM:
top - 11:02:53 up 27 days, 33 min,  3 users,  load average: 0.00, 0.00, 0.00
Tasks:  91 total,   1 running,  90 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.2%us,  0.1%sy,  0.0%ni, 99.5%id,  0.1%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   1020400k total,   881004k used,   139396k free,   162632k buffers
Swap:  1835000k total,    14984k used,  1820016k free,   220644k cached
 
http://imgur.com/blULjDa shows the hypervisor panel of Horizon. As you can see, Compute 3 has fewer resources used, but none of the compute nodes should be anywhere near overloaded from what I can tell.
 
Any ideas? Let me know if I'm missing anything obvious that would help with figuring this out! 
 
Hannah


***********

Radiant Worlds Limited is registered in England (company no: 07822337). This message is intended solely for the addressee and may contain confidential information. If you have received this message in error please send it back to us and immediately and permanently delete it from your system. Do not use, copy or disclose the information contained in this message or in any attachment. Please also note that transmission cannot be guaranteed to be secure or error-free.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140827/dfa83f09/attachment.html>