Open Stack

Hello,

I wanted to follow-up with this issue in case anyone was curious and also
for the list archives.

There was an existing bug report about this here:
https://bugs.launchpad.net/nova/+bug/1092347. We tested out the
share_dhcp_address option today and it worked well.

Joe



On Wed, Jan 15, 2014 at 5:05 PM, Joe Topjian <joe at topjian.net> wrote:

> Hello,
>
> I ran into a very odd issue today when setting up a new OpenStack cloud.
> Instances that were migrated to another compute node lost communication
> with the DHCP server once their lease was up.
>
> The cloud is configured with nova-network, FlatDHCPManager, and uses
> multi-host. Shared storage is not being used, so we were migrating with
> --block-migrate.
>
> We narrowed the issue down to iptables. The rules are acting very funny.
>
> On the source compute node (192.168.1.12), before migrating:
>
> :nova-compute-inst-49 - [0:0]
> -A nova-compute-inst-49 -m state --state INVALID -j DROP
> -A nova-compute-inst-49 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-provider
> -A nova-compute-inst-49 -s 192.168.1.12/32 -p udp -m udp --sport 67
> --dport 68 -j ACCEPT
> -A nova-compute-inst-49 -p icmp -j ACCEPT
> -A nova-compute-inst-49 -p tcp -m tcp --dport 22 -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-sg-fallback
>
> On the destination compute node (192.168.1.11), after migrating:
>
> :nova-compute-inst-49 - [0:0]
> -A nova-compute-inst-49 -m state --state INVALID -j DROP
> -A nova-compute-inst-49 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-provider
> -A nova-compute-inst-49 -s 192.168.1.12/32 -p udp -m udp --sport 67
> --dport 68 -j ACCEPT
> -A nova-compute-inst-49 -p icmp -j ACCEPT
> -A nova-compute-inst-49 -p tcp -m tcp --dport 22 -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-sg-fallback
>
> Note how 192.168.1.12 was directly copied over. The old compute node no
> longer accepts the instance's lease request and performs a DHCP NAK. This
> is now an invalid rule.
>
> After 60 seconds, the instance loses its DHCP lease and becomes
> unreachable.
>
> On the destination compute node after hard rebooting the instance:
>
> :nova-compute-inst-49 - [0:0]
> -A nova-compute-inst-49 -m state --state INVALID -j DROP
> -A nova-compute-inst-49 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-provider
> -A nova-compute-inst-49 -s 192.168.1.12/32 -p udp -m udp --sport 67
> --dport 68 -j ACCEPT
> -A nova-compute-inst-49 -p icmp -j ACCEPT
> -A nova-compute-inst-49 -p tcp -m tcp --dport 22 -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-sg-fallback
> -A nova-compute-inst-49 -s 192.168.1.11/32 -p udp -m udp --sport 67
> --dport 68 -j ACCEPT
>
> Note how 192.168.1.11 has been added to the ruleset, but it's after the
> fallback jump. The fallback jump simply drops the packet.
>
> So we were scratching our heads on what to do. The first thing we tried
> was to delete the fallback jump. That worked. But when we rebooted the
> node, the rule was, of course, reinjected.
>
> Our next thought was to add a security group rule allowing DHCP. We did
> that and saw that any edit to the security group fixed the whole issue!
>
> Note the addition of a port 80 rule, and how the DHCP rule is for the
> right server as well as in the right location:
>
> :nova-compute-inst-49 - [0:0]
> -A nova-compute-inst-49 -m state --state INVALID -j DROP
> -A nova-compute-inst-49 -m state --state RELATED,ESTABLISHED -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-provider
> -A nova-compute-inst-49 -s 192.168.1.11/32 -p udp -m udp --sport 67
> --dport 68 -j ACCEPT
> -A nova-compute-inst-49 -p icmp -j ACCEPT
> -A nova-compute-inst-49 -p tcp -m tcp --dport 22 -j ACCEPT
> -A nova-compute-inst-49 -p tcp -m tcp --dport 80 -j ACCEPT
> -A nova-compute-inst-49 -j nova-compute-sg-fallback
>
>
> Does anyone know what's going on here?
>
> Thanks,
> Joe
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20140120/fadfeee0/attachment.html>