[Openstack-operators] Networking breaks in CentOS guests but works with Ubuntu guests

Lorin Hochstein lorin at nimbisservices.com
Wed Apr 3 20:59:29 UTC 2013


That was my instinct, but I've tried it both ways (toggling
libvirt_use_virtio_for_bridge, restarting nova-compute, launching new
instance), and vnc'd into the instance to confirmed that in one case the
virtio_net drivers were loaded, and in another case, they weren't, and the
result was the same. But it doesn't seem to be related. It's really
baffling.

Lorin


On Wed, Apr 3, 2013 at 4:47 PM, Joe Topjian <joe.topjian at cybera.ca> wrote:

> That's really bizarre -- especially since it's only CentOS images. Do you
> think it might be something with virtio compatibility?
>
> I'm hesitant to lean on it being a compute/controller issue since other
> images work.
>
>
> On Wed, Apr 3, 2013 at 2:41 PM, Lorin Hochstein <lorin at nimbisservices.com>wrote:
>
>> I've tested with multiple ones, including the CentOS6 image from that
>> page, as well as several we have rolled on our own.
>>
>> Right now I'm testing by manually putting on the IP by doing:
>>
>> ip addr add 10.40.0.4/16 broadcast 10.40.255.255 dev eth0
>>
>> I can't ping out at all. If I try to arping out, and then tcpdump, just
>> like in the DHCP case, I can see the ARP request and replies on vnet0 of
>> the host:
>>
>> root at c220-2:~# tcpdump -i vnet0 arp
>> tcpdump: WARNING: vnet0: no IPv4 address assigned
>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>> 16:34:42.109067 ARP, Request who-has 10.40.0.1 (Broadcast) tell
>> 10.40.0.4, length 28
>> 16:34:42.109085 ARP, Request who-has 10.40.0.1 (Broadcast) tell
>> 10.40.0.4, length 28
>> 16:34:42.109216 ARP, Reply 10.40.0.1 is-at 54:78:1a:86:50:c9 (oui
>> Unknown), length 46
>>
>>
>> But if I tcpdump on eth0 in the guest, I only see the arp requests, not
>> the replies..
>>
>>
>> Lorin
>>
>>
>> On Wed, Apr 3, 2013 at 4:26 PM, Joe Topjian <joe.topjian at cybera.ca>wrote:
>>
>>> What CentOS images are you using? These have worked for me:
>>>
>>> https://github.com/rackerjoe/oz-image-build
>>>
>>>
>>> On Wed, Apr 3, 2013 at 2:13 PM, Lorin Hochstein <
>>> lorin at nimbisservices.com> wrote:
>>>
>>>> Hi Joe:
>>>>
>>>> It happens immediately thereafter. CentOS images have never worked on
>>>> our setup.
>>>>
>>>> Lorin
>>>>
>>>>
>>>> On Wed, Apr 3, 2013 at 3:30 PM, Joe Topjian <joe.topjian at cybera.ca>wrote:
>>>>
>>>>> Hi Lorin,
>>>>>
>>>>> Does this happen shortly after the guests were created? Or usually a
>>>>> few hours/days later? If the latter, are these guests seeing large amounts
>>>>> of bandwidth?
>>>>>
>>>>> Thanks,
>>>>> Joe
>>>>>
>>>>>
>>>>> On Wed, Apr 3, 2013 at 1:16 PM, Lorin Hochstein <
>>>>> lorin at nimbisservices.com> wrote:
>>>>>
>>>>>> Hi all:
>>>>>>
>>>>>> I'm having a strange issue where networking on my CentOS guests isn't
>>>>>> working properly, but things are working fine with my Ubuntu guests.
>>>>>>
>>>>>> I'm running Folsom on Ubuntu 12.04, nova-network, not multi-host.
>>>>>>
>>>>>> The first symptom is that CentOS instances don't get IP addresses via
>>>>>> DHCP. If I trace the DHCP requests and replies using tcpdump, I can see the
>>>>>> reply from dnsmasq reach the vnetX interface of the compute host, but it
>>>>>> doesn't get to the eth0 interface of the compute host. (I'm at a loss here
>>>>>> about how to debug something like that).
>>>>>>
>>>>>> If I try to statically configure an IP address on the guest instead,
>>>>>> networking still doesn't work. I can't ping anything on the subnet, and I
>>>>>> don't even see the icmp traffic on vnetX of the host.
>>>>>>
>>>>>> I've tried this twiddling the following options, but no change in
>>>>>> behavior:
>>>>>>
>>>>>> * Adding the following rule to nova-network node: iptables -A
>>>>>> POSTROUTING -t mangle -p udp --dport bootpc -j CHECKSUM --checksum-fill
>>>>>> * Adding the same rule to nova-compute node
>>>>>> * Setting libvirt_use_virtio_for_bridge to "yes" and "no" (restarting
>>>>>> nova-compute, re-launching instances)
>>>>>> * With and without vhost_net loaded in nova-compute (restarting
>>>>>> nova-compute, re-launching instances)
>>>>>> * Disabling iIpv6 inside of the CentOS guest
>>>>>>
>>>>>> Has anybody encountered this before?
>>>>>>
>>>>>> Lorin
>>>>>>
>>>>>> --
>>>>>> Lorin Hochstein
>>>>>> Lead Architect - Cloud Services
>>>>>> Nimbis Services, Inc.
>>>>>> www.nimbisservices.com
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-operators mailing list
>>>>>> OpenStack-operators at lists.openstack.org
>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Joe Topjian
>>>>> Systems Administrator
>>>>> Cybera Inc.
>>>>>
>>>>> www.cybera.ca
>>>>>
>>>>> Cybera is a not-for-profit organization that works to spur and
>>>>> support innovation, for the economic benefit of Alberta, through the use
>>>>> of cyberinfrastructure.
>>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> Lorin Hochstein
>>>> Lead Architect - Cloud Services
>>>> Nimbis Services, Inc.
>>>> www.nimbisservices.com
>>>>
>>>
>>>
>>>
>>> --
>>> Joe Topjian
>>> Systems Administrator
>>> Cybera Inc.
>>>
>>> www.cybera.ca
>>>
>>> Cybera is a not-for-profit organization that works to spur and support
>>> innovation, for the economic benefit of Alberta, through the use
>>> of cyberinfrastructure.
>>>
>>
>>
>>
>> --
>> Lorin Hochstein
>> Lead Architect - Cloud Services
>> Nimbis Services, Inc.
>> www.nimbisservices.com
>>
>
>
>
> --
> Joe Topjian
> Systems Administrator
> Cybera Inc.
>
> www.cybera.ca
>
> Cybera is a not-for-profit organization that works to spur and support
> innovation, for the economic benefit of Alberta, through the use
> of cyberinfrastructure.
>



-- 
Lorin Hochstein
Lead Architect - Cloud Services
Nimbis Services, Inc.
www.nimbisservices.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130403/c2b83342/attachment.html>


More information about the OpenStack-operators mailing list