Hi all,<div><br></div><div>I have fixed my problem.</div><div><br></div><div>Two things:</div><div>1) There was a missing route from VM instance network to metadata service through public network. So after adding the rule on phys router I got that working.</div>
<div>2) Quantum auth settings were not configured inside nova.conf on the box where the nova-api is running, so metadata service couldn't connect to quantum and was failing.</div><div><br></div><div><br><div class="gmail_quote">
On Wed, Oct 3, 2012 at 11:01 PM, Jānis Ģeņģeris <span dir="ltr"><<a href="mailto:janis.gengeris@gmail.com" target="_blank">janis.gengeris@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="gmail_quote"><div><div class="h5">On Tue, Oct 2, 2012 at 5:55 PM, Dan Wendlandt <span dir="ltr"><<a href="mailto:dan@nicira.com" target="_blank">dan@nicira.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Janis,<br>
<br>
Thanks for the detailed report. Responses inline.<br>
<br>
dan<br>
<div><br>
On Tue, Oct 2, 2012 at 7:13 AM, Jānis Ģeņģeris <<a href="mailto:janis.gengeris@gmail.com" target="_blank">janis.gengeris@gmail.com</a>> wrote:<br>
> Hello all,<br>
><br>
> I'm trying to set up quantum+openvswitch with the Folsom release. The<br>
> intended configuration is fixed IP network <a href="http://10.0.1.0/24" target="_blank">10.0.1.0/24</a> and floating IP<br>
> network <a href="http://85.254.50.0/24" target="_blank">85.254.50.0/24</a>. And am a little stuck with connection problems to<br>
> VMs.<br>
><br>
> My config is the following:<br>
><br>
> 1) Controller node that is running rabbit, mysql, quantum-server, nova-api,<br>
> nova-scheduler, nova-volume, keystone, etc. Have two net interfaces, one for<br>
> service network (192.168.164.1) and other for outside world connections.<br>
><br>
> 2) Compute node, which is working also as quantum network node, and is<br>
> running: kvm, nova-compute, quantum-l3-agent, quantum-dchp-agent. Have two<br>
> net interfaces, one is from service network 192.168.164.101, and the other<br>
> is for floating ips <a href="http://85.254.50.0/24" target="_blank">85.254.50.0/24</a>, bridged into openvswitch. And using<br>
> libvirt 0.9.11.<br>
<br>
</div>That all makes sense.<br>
<div><br>
><br>
> I wonder if local_ip in ovs_quantum_plugin.ini might break something,<br>
> because the docs say that it should be set only on hypervisors, but I have<br>
> merged hypervisor with network node.<br>
><br>
> ovs_quantum_plugin.ini fragment:<br>
> [OVS]<br>
> enable_tunneling = True<br>
> tenant_network_type = gre<br>
> tunnel_id_ranges = 1:1000<br>
> local_ip = 192.168.164.101<br>
<br>
</div>that should be fine. besides, the communication that is not working<br>
is all within one device, based on your description.<br>
<div><br>
><br>
> nova.conf fragment:<br>
> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchVirtualPortDriver<br>
> libvirt_use_virtio_for_bridges=True<br>
><br>
> The VMs are getting created successfully, nova-compute.log and console-log<br>
> for each vm looks ok.<br>
><br>
> Here are the dumps of current network configuration:<br>
><br>
> ovs-vsctl show - <a href="http://pastebin.com/0V6kRw1N" target="_blank">http://pastebin.com/0V6kRw1N</a><br>
> ip addr (on default namespace) - <a href="http://pastebin.com/VTLbit11" target="_blank">http://pastebin.com/VTLbit11</a><br>
> output from router and dhcp namespaces - <a href="http://pastebin.com/pDmjpmLE" target="_blank">http://pastebin.com/pDmjpmLE</a><br>
><br>
> pings for gateways in router namespace work ok:<br>
> # ip netns exec qrouter-3442d231-2e00-4d26-823e-1feb5d02a798 ping 10.0.1.1<br>
> # ip netns exec qrouter-3442d231-2e00-4d26-823e-1feb5d02a798 ping<br>
> 85.254.50.1<br>
><br>
> But it is not possible to ping any of the instances in fixed network from<br>
> router namespace (floating network is also not working of course).<br>
><br>
> a) Can this be an iptables/NAT problem?<br>
> b) What about libvirt nwfilters, they are also active.<br>
<br>
</div>unlikely, given that you're using the integrated OVS vif driver, which<br>
doesn't invoke iptables hooks.<br>
<div><br>
> c) What else could be wrong?<br>
<br>
<br>
</div>Two questions:<br>
<br>
1) have you confirmed that the VMs got an IP via DHCP? You can do<br>
this by looking at the console log, or by using VNC to access the<br>
instances.<br></blockquote></div></div><div>I think this excerpt from vm console-log means a big no:</div><div><br></div><div><div>cloud-init start-local running: Tue, 02 Oct 2012 13:36:32 +0000. up 4.45 seconds</div><div>
no instance data found in start-local</div>
<div>cloud-init-nonet waiting 120 seconds for a network device.</div><div>cloud-init-nonet gave up waiting for a network device.</div><div>ci-info: lo : 1 127.0.0.1 255.0.0.0 .</div><div>ci-info: eth0 : 1 . . fa:16:3e:61:7b:bc</div>
<div>route_info failed</div></div><div class="im"><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
2) if so, can you confirm that you can ping the DHCP IP address in the<br>
subnet from the router namespace?</blockquote></div><div>See answer to 1).</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"> </blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It would also be good to run tcpdump on the linux device that<br>
corresponds to the VM you are pinging (i.e., vnetX for VM X).<br></blockquote></div><div>Running tcpdump on any of the vnetX interfaces can see any ping request to the IPs in fixed subnet <a href="http://10.0.1.0/24" target="_blank">10.0.1.0/24</a>.</div>
<div>For example:</div><div># tcpdump -i vnet3</div><div><div>tcpdump: WARNING: vnet3: no IPv4 address assigned</div><div>tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div><div>listening on vnet3, link-type EN10MB (Ethernet), capture size 65535 bytes</div>
<div>22:03:08.565579 ARP, Request who-has 10.0.1.6 tell 10.0.1.1, length 28</div><div>22:03:09.564696 ARP, Request who-has 10.0.1.6 tell 10.0.1.1, length 28</div></div><div><br></div><div>The only responsive IP is the on set on the gw.</div>
<div class="im">
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Its possible this is related to the specific vif-driver, which is<br>
using the new libvirt integrated support for OVS.<br>
<br>
For example, you could try:<br>
<br>
libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver<br>
<br>
but remember that when you do this, you will also want to open up the<br>
default security group:<br>
<br>
nova secgroup-add-rule default icmp -1 -1 <a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a></blockquote></div><div>Haven't tried this part yet, will do tomorrow.</div><div><br></div><div>But looks like I have problem with metadata-api service. I changed metadata service address inside l3-agent config to public one 85.254.49.100.</div>
<div><br></div><div>I have this iptables rule inside router namespace now:</div><div><div><div>Chain quantum-l3-agent-PREROUTING (1 references)</div><div> pkts bytes target prot opt in out source destination </div>
<div> 0 0 DNAT tcp -- * * <a href="http://0.0.0.0/0" target="_blank">0.0.0.0/0</a> 169.254.169.254 tcp dpt:80 to:<a href="http://85.254.49.100:8775" target="_blank">85.254.49.100:8775</a></div>
</div></div><div><br>
</div><div>But with 0 hits anyway.</div><div><br></div><div>The metadata service is now accessible if going directly to 85.254.49.100 from router namesapce.</div><div>I think what I' missing is the route from VM network on metadata host as described here</div>
<div><a href="http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html" target="_blank">http://docs.openstack.org/trunk/openstack-network/admin/content/adv_cfg_l3_agent_metadata.html</a>,</div>
<div>except I have no clue how to set and get it working. </div>
<div><br></div><div>When trying:</div><div># ip netns exec qrouter-3442d231-2e00-4d26-823e-1feb5d02a798 nc -v 169.254.169.254 80</div><div><div>nc: connect to 169.254.169.254 port 80 (tcp) failed: Connection refused</div>
</div><div><br></div><div>How will private IP from <a href="http://10.0.1.0/24" target="_blank">10.0.1.0/24</a> get to the metada-service through public network?</div><div class="im"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Dan<br>
<div><br>
<br>
><br>
> Any help and comments how to fix this are welcome.<br>
><br>
> Regards,<br>
> --janis<br>
><br>
</div>> _______________________________________________<br>
> OpenStack-operators mailing list<br>
> <a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br>
><br>
<span><font color="#888888"><br>
<br>
<br>
--<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
Dan Wendlandt<br>
Nicira, Inc: <a href="http://www.nicira.com" target="_blank">www.nicira.com</a><br>
twitter: danwendlandt<br>
~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
</font></span></blockquote></div></div>Oh I'm little confused with all the network namespaces and vswitch in the middle. Can someone who have real running setup explain briefly what is required for basic setup to start working, as I see I'm not the only, nor the last one having this problem.<span class="HOEnZb"><font color="#888888"><br>
<br clear="all"><div><br></div>-- <br>--janis<br>
</font></span></blockquote></div><br><br clear="all"><div><br></div>-- <br>--janis<br>
</div>