Hi all, I'm hoping I could get some assistance figuring out my networking problems with a small Essex test cluster. I have a small Diablo cluster running without any problems but have hit a wall in deploying Essex.<div>
<br></div><div>I can launch VMs without issue and access them from the compute host, but from there I can't access anything except the host, DNS services, and other VMs.<br>
<div><br></div><div>I have separate machines running keystone, glance, postgresql, rabbit-mq and nova-api. They're all on the .os domain with 172.22.1.X IPs</div><div><br></div><div>I have one machine running nova-compute, nova-network and nova-api, with a public address 192.43.239.175 and also an IP on the 172.22.1.X subnet in the .os domain. It has the following nova/conf:</div>
<div><br></div><div><div>--dhcpbridge_flagfile=/etc/nova/nova.conf</div><div>--dhcpbridge=/usr/bin/nova-dhcpbridge</div><div>--logdir=/var/log/nova</div><div>--state_path=/var/lib/nova</div><div>--lock_path=/var/lock/nova</div>
<div>--force_dhcp_release</div><div>--iscsi_helper=tgtadm</div><div>--libvirt_use_virtio_for_bridges</div><div>--connection_type=libvirt</div><div>--root_helper=sudo nova-rootwrap</div><div>--verbose</div><div>--ec2_private_dns_show_ip</div>
<div><br></div><div>--network_manager=nova.network.manager.FlatDHCPManager</div><div>--rabbit_host=os-amqp.os</div><div>--sql_connection=postgresql://[user]:[password]@os-sql.os/nova</div><div>--image_service=nova.image.glance.GlanceImageService</div>
<div>--glance_api_servers=os-glance.os:9292</div><div>--auth_strategy=keystone</div><div>--scheduler_driver=nova.scheduler.simple.SimpleScheduler</div><div>--keystone_ec2_url=<a href="http://os-key.os:5000/v2.0/ec2tokens" target="_blank">http://os-key.os:5000/v2.0/ec2tokens</a></div>
<div><br></div><div>--api_paste_config=/etc/nova/api-paste.ini</div><div><br></div><div>--my_ip=192.43.239.175</div><div>--flat_interface=eth0</div><div>--public_interface=eth1</div><div>--multi_host=True</div><div>--routing_source_ip=192.43.239.175</div>
<div>--network_host=192.43.239.175</div><div><br></div><div>--dmz_cidr=$my_ip</div><div><br></div><div>--ec2_host=192.43.239.175</div><div>--ec2_dmz_host=192.43.239.175</div><div><br></div><div>I believe I'm seeing a natting issue of some sort - my VMs cannot ping external IPs, though DNS seems to work.</div>
<div><div>ubuntu@monday:~$ ping <a href="http://www.google.com" target="_blank">www.google.com</a></div><div>PING <a href="http://www.l.google.com" target="_blank">www.l.google.com</a> (74.125.237.148) 56(84) bytes of data.</div>
</div><div><AWKWARD SILENCE></div>
<div><br></div><div>When I do a tcpdump on the compute host things seem fairly normal, even though nothing is getting back to the VM</div><div><div><br></div><div>root@ncios1:~# tcpdump icmp -i br100</div><div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div>
<div>listening on br100, link-type EN10MB (Ethernet), capture size 65535 bytes</div><div>14:35:28.046416 IP 10.0.0.8 > <a href="http://syd01s13-in-f20.1e100.net" target="_blank">syd01s13-in-f20.1e100.net</a>: ICMP echo request, id 5002, seq 9, length 64</div>
<div>14:35:28.051477 IP <a href="http://syd01s13-in-f20.1e100.net" target="_blank">syd01s13-in-f20.1e100.net</a> > <a href="http://10.0.0.8" target="_blank">10.0.0.8</a>: ICMP echo reply, id 5002, seq 9, length 64</div>
<div>14:35:29.054505 IP 10.0.0.8 > <a href="http://syd01s13-in-f20.1e100.net" target="_blank">syd01s13-in-f20.1e100.net</a>: ICMP echo request, id 5002, seq 10, length 64</div>
<div>14:35:29.059556 IP <a href="http://syd01s13-in-f20.1e100.net" target="_blank">syd01s13-in-f20.1e100.net</a> > <a href="http://10.0.0.8" target="_blank">10.0.0.8</a>: ICMP echo reply, id 5002, seq 10, length 64</div>
</div><div><br></div><div>I've pored over the iptables nat rules and can't see anything amiss apart from the masquerades that are automatically added: (I've cut out some empty chains for brevity)</div>
<div><br></div><div><div>root@ncios1:~# iptables -L -t nat -v</div><div>Chain PREROUTING (policy ACCEPT 22 packets, 2153 bytes)</div><div> pkts bytes target prot opt in out source destination </div>
<div> 22 2153 nova-network-PREROUTING all -- any any anywhere anywhere </div><div> 22 2153 nova-compute-PREROUTING all -- any any anywhere anywhere </div>
<div> 22 2153 nova-api-PREROUTING all -- any any anywhere anywhere </div><div><br></div><div>Chain INPUT (policy ACCEPT 12 packets, 1573 bytes)</div><div> pkts bytes target prot opt in out source destination </div>
<div><br></div><div>Chain OUTPUT (policy ACCEPT 31 packets, 2021 bytes)</div><div> pkts bytes target prot opt in out source destination </div><div> 31 2021 nova-network-OUTPUT all -- any any anywhere anywhere </div>
<div> 31 2021 nova-compute-OUTPUT all -- any any anywhere anywhere </div><div> 31 2021 nova-api-OUTPUT all -- any any anywhere anywhere </div><div><br>
</div><div>Chain POSTROUTING (policy ACCEPT 30 packets, 1961 bytes)</div><div> pkts bytes target prot opt in out source destination </div><div> 31 2021 nova-network-POSTROUTING all -- any any anywhere anywhere </div>
<div> 30 1961 nova-compute-POSTROUTING all -- any any anywhere anywhere </div><div> 30 1961 nova-api-POSTROUTING all -- any any anywhere anywhere </div>
<div> 30 1961 nova-postrouting-bottom all -- any any anywhere anywhere </div><div> 0 0 MASQUERADE tcp -- any any <a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> !<a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> masq ports: 1024-65535</div>
<div> 0 0 MASQUERADE udp -- any any <a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> !<a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> masq ports: 1024-65535</div>
<div> 0 0 MASQUERADE all -- any any <a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> !<a href="http://192.168.122.0/24" target="_blank">192.168.122.0/24</a> </div>
</div><div><div><br></div><div>Chain nova-api-snat (1 references)</div><div> pkts bytes target prot opt in out source destination </div><div> 30 1961 nova-api-float-snat all -- any any anywhere anywhere </div>
<div><br></div><div>Chain nova-compute-snat (1 references)</div><div> pkts bytes target prot opt in out source destination </div><div> 30 1961 nova-compute-float-snat all -- any any anywhere anywhere </div>
<div><br></div><div>Chain nova-network-POSTROUTING (1 references)</div><div> pkts bytes target prot opt in out source destination </div><div> 0 0 ACCEPT all -- any any <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a> <a href="http://nri5.nci.org.au" target="_blank">nri5.nci.org.au</a> </div>
<div> 0 0 ACCEPT all -- any any <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a> <a href="http://nri5.nci.org.au" target="_blank">nri5.nci.org.au</a> </div><div> 1 60 ACCEPT all -- any any <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a> <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a> ! ctstate DNAT</div>
<div><br></div><div>Chain nova-network-PREROUTING (1 references)</div><div> pkts bytes target prot opt in out source destination </div><div> 0 0 DNAT tcp -- any any anywhere 169.254.169.254 tcp dpt:http to:<a href="http://192.43.239.175:8775" target="_blank">192.43.239.175:8775</a></div>
<div><br></div><div>Chain nova-network-snat (1 references)</div><div> pkts bytes target prot opt in out source destination </div><div> 30 1961 nova-network-float-snat all -- any any anywhere anywhere </div>
<div> 0 0 SNAT all -- any any <a href="http://10.0.0.0/8" target="_blank">10.0.0.0/8</a> anywhere to:192.43.239.175</div><div><br></div><div>Chain nova-postrouting-bottom (1 references)</div>
<div>
pkts bytes target prot opt in out source destination </div><div> 30 1961 nova-network-snat all -- any any anywhere anywhere </div><div> 30 1961 nova-compute-snat all -- any any anywhere anywhere </div>
<div> 30 1961 nova-api-snat all -- any any anywhere anywhere </div></div><div><br></div><div>and the ACCEPT icmp rule seems to be there in filter for the security group as well, though it's not being triggered for some reason:</div>
<div><br></div>
<div><div>Chain nova-compute-inst-6 (1 references)</div><div> pkts bytes target prot opt in out source destination </div><div> 0 0 DROP all -- any any anywhere anywhere state INVALID</div>
<div> 39 6545 ACCEPT all -- any any anywhere anywhere state RELATED,ESTABLISHED</div><div> 1 60 nova-compute-provider all -- any any anywhere anywhere </div>
<div> 0 0 ACCEPT udp -- any any 10.0.0.3 anywhere udp spt:bootps dpt:bootpc</div><div> 1 60 ACCEPT all -- any any <a href="http://10.0.0.0/24" target="_blank">10.0.0.0/24</a> anywhere </div>
<div> 0 0 ACCEPT icmp -- any any anywhere anywhere </div><div> 0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:ssh</div><div> 0 0 nova-compute-sg-fallback all -- any any anywhere anywhere </div>
</div><div><br></div><div>I've tried changing the routing source IP between using the private 172.22.1.X IP and the public one but it doesn't seem to change anything. I tried without that config option at all and also without the network host flag and not much seems to change.</div>
<div><br></div><div>Any help would be much appreciated.</div><div><br></div><div><br></div><div><br></div>-- <br><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><div><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">Michael Chapman</span></div>
<div><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px"><i>Cloud Computing Services</i></span></div><div><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">ANU Supercomputer Facility<br>
Room 318, Leonard Huxley Building (#56), Mills Road<br>The Australian National University<br>Canberra ACT 0200 Australia</span></div><div><span style="border-collapse:collapse;font-family:arial,sans-serif;font-size:13px">Tel: <i><a href="tel:%2B61%202%206125%C2%A07106" value="+61261257106" target="_blank">+61 2 6125 7106</a></i><br>
Web: <a href="http://nci.org.au" target="_blank">http://nci.org.au</a></span></div></span><br>
</div></div>