[Openstack-operators] Networking breaks in CentOS guests but works with Ubuntu guests

Narayan Desai narayan.desai at gmail.com
Thu Apr 4 15:33:58 UTC 2013


iirc, vhost_net is only needed on the host.

We have seen stability issues with 12.04 (only on particular host
types) when using virtio without vhost_net. Enabling vhost_net on the
host resolved the issues for us.

Which version of Centos are you running?
 -nld

On Wed, Apr 3, 2013 at 3:59 PM, Lorin Hochstein
<lorin at nimbisservices.com> wrote:
> That was my instinct, but I've tried it both ways (toggling
> libvirt_use_virtio_for_bridge, restarting nova-compute, launching new
> instance), and vnc'd into the instance to confirmed that in one case the
> virtio_net drivers were loaded, and in another case, they weren't, and the
> result was the same. But it doesn't seem to be related. It's really
> baffling.
>
> Lorin
>
>
> On Wed, Apr 3, 2013 at 4:47 PM, Joe Topjian <joe.topjian at cybera.ca> wrote:
>>
>> That's really bizarre -- especially since it's only CentOS images. Do you
>> think it might be something with virtio compatibility?
>>
>> I'm hesitant to lean on it being a compute/controller issue since other
>> images work.
>>
>>
>> On Wed, Apr 3, 2013 at 2:41 PM, Lorin Hochstein <lorin at nimbisservices.com>
>> wrote:
>>>
>>> I've tested with multiple ones, including the CentOS6 image from that
>>> page, as well as several we have rolled on our own.
>>>
>>> Right now I'm testing by manually putting on the IP by doing:
>>>
>>> ip addr add 10.40.0.4/16 broadcast 10.40.255.255 dev eth0
>>>
>>> I can't ping out at all. If I try to arping out, and then tcpdump, just
>>> like in the DHCP case, I can see the ARP request and replies on vnet0 of the
>>> host:
>>>
>>> root at c220-2:~# tcpdump -i vnet0 arp
>>> tcpdump: WARNING: vnet0: no IPv4 address assigned
>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>> decode
>>> 16:34:42.109067 ARP, Request who-has 10.40.0.1 (Broadcast) tell
>>> 10.40.0.4, length 28
>>> 16:34:42.109085 ARP, Request who-has 10.40.0.1 (Broadcast) tell
>>> 10.40.0.4, length 28
>>> 16:34:42.109216 ARP, Reply 10.40.0.1 is-at 54:78:1a:86:50:c9 (oui
>>> Unknown), length 46
>>>
>>>
>>> But if I tcpdump on eth0 in the guest, I only see the arp requests, not
>>> the replies..
>>>
>>>
>>> Lorin
>>>
>>>
>>> On Wed, Apr 3, 2013 at 4:26 PM, Joe Topjian <joe.topjian at cybera.ca>
>>> wrote:
>>>>
>>>> What CentOS images are you using? These have worked for me:
>>>>
>>>> https://github.com/rackerjoe/oz-image-build
>>>>
>>>>
>>>> On Wed, Apr 3, 2013 at 2:13 PM, Lorin Hochstein
>>>> <lorin at nimbisservices.com> wrote:
>>>>>
>>>>> Hi Joe:
>>>>>
>>>>> It happens immediately thereafter. CentOS images have never worked on
>>>>> our setup.
>>>>>
>>>>> Lorin
>>>>>
>>>>>
>>>>> On Wed, Apr 3, 2013 at 3:30 PM, Joe Topjian <joe.topjian at cybera.ca>
>>>>> wrote:
>>>>>>
>>>>>> Hi Lorin,
>>>>>>
>>>>>> Does this happen shortly after the guests were created? Or usually a
>>>>>> few hours/days later? If the latter, are these guests seeing large amounts
>>>>>> of bandwidth?
>>>>>>
>>>>>> Thanks,
>>>>>> Joe
>>>>>>
>>>>>>
>>>>>> On Wed, Apr 3, 2013 at 1:16 PM, Lorin Hochstein
>>>>>> <lorin at nimbisservices.com> wrote:
>>>>>>>
>>>>>>> Hi all:
>>>>>>>
>>>>>>> I'm having a strange issue where networking on my CentOS guests isn't
>>>>>>> working properly, but things are working fine with my Ubuntu guests.
>>>>>>>
>>>>>>> I'm running Folsom on Ubuntu 12.04, nova-network, not multi-host.
>>>>>>>
>>>>>>> The first symptom is that CentOS instances don't get IP addresses via
>>>>>>> DHCP. If I trace the DHCP requests and replies using tcpdump, I can see the
>>>>>>> reply from dnsmasq reach the vnetX interface of the compute host, but it
>>>>>>> doesn't get to the eth0 interface of the compute host. (I'm at a loss here
>>>>>>> about how to debug something like that).
>>>>>>>
>>>>>>> If I try to statically configure an IP address on the guest instead,
>>>>>>> networking still doesn't work. I can't ping anything on the subnet, and I
>>>>>>> don't even see the icmp traffic on vnetX of the host.
>>>>>>>
>>>>>>> I've tried this twiddling the following options, but no change in
>>>>>>> behavior:
>>>>>>>
>>>>>>> * Adding the following rule to nova-network node: iptables -A
>>>>>>> POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill
>>>>>>> * Adding the same rule to nova-compute node
>>>>>>> * Setting libvirt_use_virtio_for_bridge to "yes" and "no" (restarting
>>>>>>> nova-compute, re-launching instances)
>>>>>>> * With and without vhost_net loaded in nova-compute (restarting
>>>>>>> nova-compute, re-launching instances)
>>>>>>> * Disabling iIpv6 inside of the CentOS guest
>>>>>>>
>>>>>>> Has anybody encountered this before?
>>>>>>>
>>>>>>> Lorin
>>>>>>>
>>>>>>> --
>>>>>>> Lorin Hochstein
>>>>>>> Lead Architect - Cloud Services
>>>>>>> Nimbis Services, Inc.
>>>>>>> www.nimbisservices.com
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-operators mailing list
>>>>>>> OpenStack-operators at lists.openstack.org
>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Joe Topjian
>>>>>> Systems Administrator
>>>>>> Cybera Inc.
>>>>>>
>>>>>> www.cybera.ca
>>>>>>
>>>>>> Cybera is a not-for-profit organization that works to spur and support
>>>>>> innovation, for the economic benefit of Alberta, through the use of
>>>>>> cyberinfrastructure.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Lorin Hochstein
>>>>> Lead Architect - Cloud Services
>>>>> Nimbis Services, Inc.
>>>>> www.nimbisservices.com
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Joe Topjian
>>>> Systems Administrator
>>>> Cybera Inc.
>>>>
>>>> www.cybera.ca
>>>>
>>>> Cybera is a not-for-profit organization that works to spur and support
>>>> innovation, for the economic benefit of Alberta, through the use of
>>>> cyberinfrastructure.
>>>
>>>
>>>
>>>
>>> --
>>> Lorin Hochstein
>>> Lead Architect - Cloud Services
>>> Nimbis Services, Inc.
>>> www.nimbisservices.com
>>
>>
>>
>>
>> --
>> Joe Topjian
>> Systems Administrator
>> Cybera Inc.
>>
>> www.cybera.ca
>>
>> Cybera is a not-for-profit organization that works to spur and support
>> innovation, for the economic benefit of Alberta, through the use of
>> cyberinfrastructure.
>
>
>
>
> --
> Lorin Hochstein
> Lead Architect - Cloud Services
> Nimbis Services, Inc.
> www.nimbisservices.com
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>



More information about the OpenStack-operators mailing list