<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hi all,</p>
<p>I've been struggling with instances suddenly not being able to
fetch metadata from Openstack Queens (this has worked fine
earlier). <br>
</p>
<p>Newly created VMs fail to connect to the magic ip, eg.
<a class="moz-txt-link-freetext" href="http://169.254.169.254/">http://169.254.169.254/</a>, and won't initialize properly.
Subsequently ssh login will fail since no key is uploaded.</p>
<p>The symptom is failed requests in the log<br>
</p>
<pre style="color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration-style: initial; text-decoration-color: initial; overflow-wrap: break-word; white-space: pre-wrap;"><b>Cirros:</b>
Starting network...
udhcpc (v1.20.1) started
Sending discover...
Sending select for 10.0.0.18...
Lease of 10.0.0.18 obtained, lease time 86400
route: SIOCADDRT: File exists
WARN: failed: route add -net "0.0.0.0/0" gw "10.0.0.1"
cirros-ds 'net' up at 0.94
checking <a class="moz-txt-link-freetext" href="http://169.254.169.254/2009-04-04/instance-id">http://169.254.169.254/2009-04-04/instance-id</a>
failed 1/20: up 0.94. request failed
failed 2/20: up 3.01. request failed
failed 3/20: up 5.03. request failed
failed 4/20: up 7.04. request failed
<b>..and on Centos6:</b>
ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
ci-info: | 0 | 169.254.169.254 | 10.0.0.1 | 255.255.255.255 | eth0 | UGH |
ci-info: | 1 | 10.0.0.0 | 0.0.0.0 | 255.255.255.0 | eth0 | U |
ci-info: | 2 | 0.0.0.0 | 10.0.0.1 | 0.0.0.0 | eth0 | UG |
ci-info: +-------+-----------------+----------+-----------------+-----------+-------+
2018-11-06 08:10:07,892 - url_helper.py[WARNING]: Calling '<a class="moz-txt-link-freetext" href="http://169.254.169.254/2009-04-04/meta-data/instance-id">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [0/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
2018-11-06 08:10:08,906 - url_helper.py[WARNING]: Calling '<a class="moz-txt-link-freetext" href="http://169.254.169.254/2009-04-04/meta-data/instance-id">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [1/120s]: unexpected error ['NoneType' object has no attribute 'status_code']
2018-11-06 08:10:09,925 - url_helper.py[WARNING]: Calling '<a class="moz-txt-link-freetext" href="http://169.254.169.254/2009-04-04/meta-data/instance-id">http://169.254.169.254/2009-04-04/meta-data/instance-id</a>' failed [2/120s]: unexpected error ['NoneType' object has no attribute
...
</pre>
<p><font size="-1">Using Curl manually, eg. '<i>curl
<a class="moz-txt-link-freetext" href="http://169.254.169.254/openstack">http://169.254.169.254/openstack</a></i>' one gets: <br>
</font></p>
<p><font size="-1"><i>curl: (52) Empty reply from server</i></font></p>
<p><font size="-1"><b>At the same time this error is showing up in
the syslog on the controller:</b></font></p>
<p>Nov 6 12:51:01 controller neutron-metadata-agent[3094]: File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/hub.py",
line 460, in fire_timers<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]:
timer()<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]: File
"/usr/local/lib/python2.7/dist-packages/eventlet/hubs/timer.py",
line 59, in __call__<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]:
cb(*args, **kw)<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]: File
"/usr/local/lib/python2.7/dist-packages/eventlet/greenthread.py",
line 219, in main<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]:
result = function(*args, **kwargs)<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]: File
"/usr/local/lib/python2.7/dist-packages/eventlet/wsgi.py", line
793, in process_request<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]:
proto.__init__(conn_state, self)<br>
Nov 6 12:51:01 controller neutron-metadata-agent[3094]:
TypeError: __init__() takes exactly 4 arguments (3 given)</p>
<p><font size="-1"><b>Neither rebooting the controller, reinstalling
neutron, or restarting the services will do anything top fix
this.</b></font><br>
</p>
<p>Has anyone else seen this? We are using Queens with a single
controller.</p>
<p>Kind Regards</p>
<p>Terje Lundin</p>
<p> <br>
</p>
<p><br>
</p>
<p><br>
</p>
</body>
</html>