Hmmm, I did do some playing with security groups, but even the last network (network5) that I created was unchanged from "default" - maybe it has something to do with that, I'm testing a brand new tenant/project now with a similar setup, except I'm setting up all the networking / networks first this time.<br><div><br></div><div><br></div><br><div class="gmail_quote">On Thu Jan 15 2015 at 17:12:47 Edgar Magana <<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Did you change the security groups? </div>
<div>I will try to compare the iptables configuration for each name space</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Alex Leonhardt <<a href="mailto:aleonhardt.py@gmail.com" target="_blank">aleonhardt.py@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, January 15, 2015 at 9:03 AM<br>
<span style="font-weight:bold">To: </span>Edgar Magana <<a href="mailto:edgar.magana@workday.com" target="_blank">edgar.magana@workday.com</a>>, openstack-operators <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] metadata-api 500 errors<br>
</div></span></div><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><span>
<div><br>
</div>
<div>
<div>Hi Edgar,<br>
<br>
that's the crazy thing - so all the gre tunnels are up, I can see them in openvswitch and also can see that there are some openflow rules applied. I've craeted VMs on every hypervisor (including the controller, as it's a test install) on <span style="line-height:1.5">network1
(192.168.1.0), every VM (and that is still the case now) started there works just fine and gets the metadata as expected, the same for network2 (192.168.2.0).</span>
<div><br>
</div>
<div>the issue only appeared after I created network3 (192.168.3.0), VMs there (tried again all 3 hypervisors) get a 500 errror instead of the expected metadata files/json. The same for any / all other networks I created after (network4 and 5).</div>
<div><br>
</div>
<div>On the VM all I can see is this: </div>
<div>
<pre style="padding:8.5px;font-family:Menlo,Monaco,'Courier New',monospace;font-size:12px;color:rgb(51,51,51);border-radius:4px;margin-top:0px;margin-bottom:9px;line-height:18px;border:1px solid rgba(0,0,0,0.14902);white-space:pre-wrap;word-break:break-all;word-wrap:break-word;background-color:rgb(245,245,245)">2015-01-15 17:02:57,310 - url_helper.py[WARNING]: Calling '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [0/120s]: bad status code [500]
2015-01-15 17:02:58,509 - url_helper.py[WARNING]: Calling '<a href="http://169.254.169.254/2009-04-04/meta-data/instance-id" target="_blank">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [1/120s]: bad status code [500]</pre>
</div>
<div>Alex<br>
<div><br>
</div>
</div>
<br>
<div class="gmail_quote">On Thu Jan 15 2015 at 16:53:31 Edgar Magana <<a href="mailto:edgar.magana@workday.com" target="_blank">edgar.magana@workday.com</a>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Alex,</div>
<div><br>
</div>
<div>Did you follow the networking recommendations:</div>
<div><a href="http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html" target="_blank">http://docs.openstack.org/openstack-ops/content/network_troubleshooting.html</a></div>
<div><br>
</div>
<div>It will ell you if you write your own topology and complete a packet trace to find out the issue. </div>
<div>Make sure all tunnels are established between your three nodes.</div>
<div><br>
</div>
<div>Thanks,</div>
<div><br>
</div>
<div>Edgar</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt">
<span style="font-weight:bold">From: </span>Alex Leonhardt <<a href="mailto:aleonhardt.py@gmail.com" target="_blank">aleonhardt.py@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Thursday, January 15, 2015 at 7:45 AM<br>
<span style="font-weight:bold">To: </span>openstack-operators <<a href="mailto:openstack-operators@lists.openstack.org" target="_blank">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>[Openstack-operators] metadata-api 500 errors<br>
</div>
</span></div>
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<span>
<div><br>
</div>
<div>
<div>hi,
<div><br>
</div>
<div>i've got a test openstack install with 3 nodes, using gre tunneling -- </div>
<div><br>
</div>
<div>initially it all worked fine, but, after creating > 2 networks, VMs in networks 3,4,5 do not seem to get the metadata due to it erroring with 500 errors. whilst this is happening, VMs in networks 1 and 2 are still working fine and can be provisioned OK. </div>
<div><br>
</div>
<div>anyone seen something similar or ideas on how to go about troubleshooting this ? I got a tcpdump from the VM but as it does get to the metadata api, am not sure where the issue is (especially since other VMs in other Networks work just fine)</div>
<div><br>
</div>
<div>any ideas ? </div>
<div><br>
</div>
<div>Alex</div>
<div><br>
</div>
</div>
</div>
</span></div>
</blockquote>
</div>
</div>
</div>
</span></div></blockquote></div>