<div dir="ltr"><div class="gmail_default" style="font-family:'times new roman',serif">This is kinda hard to describe. I am having problems with dashboard.</div><div class="gmail_default" style="font-family:'times new roman',serif">
<br></div><div class="gmail_default" style="font-family:'times new roman',serif">I am running on Centos 6.5</div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">
Dashboard is installed on my controller node. When I first installed dashboard and tested, I was able to login as "jeff" but then getting 500 INTERNAL SERVER ERROR responses with both Mozilla Firefox and Google Chrome. I looked through /var/log/error_log and found that the problem was a ConnectionError in /usr/lib/python2.6/site-packages/requests/adapters.py line 206. I never resolved that issue. I sent a message to the mailing list at python-requests, which is the project that write adapters.py asking them for more troubleshooting information from the exception handler.</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">I rebooted the controller node to test if the services would come back. openstack-nova-api failed to start because:<br>
<pre>2014-07-30 13:28:52.356 4651 ERROR nova.wsgi [-] Could not bind to <a href="http://0.0.0.0:8775">0.0.0.0:8775</a><br>2014-07-30 13:28:52.357 4651 CRITICAL nova [-] error: [Errno 98] Address already in use</pre>nova-api-metadata has already grabbed port 8775. I don't understand why nova processes are running on my controller when they are compute processes, but then there is a lot about openstack that I don't understand (and I appreciate your bearing with me).<br>
<br></div><div><div class="gmail_default" style="font-family:'times new roman',serif">I am back to troubleshooting dashboard, and here is where it gets weird. It doesn't work, but it doesn't work differently depending on whether I use Google Chrome (<span style="color:rgb(48,57,66);font-family:Cantarell,Arial,sans-serif;font-size:12px">Version 36.0.1985.125</span> on Fedora 20) or Mozilla Firefox (version 24.7.0-1.el6.centos). With firefox, I get a login page. When I enter my username and password, I get a login failure. error_log says [Thu Jul 31 18:57:27 2014] [error] Login failed for user "jeff". By way of contrast, when I use Chrome, I get a 500 INTERNAL SERVER ERROR. error_log says</div>
<div><div class="gmail_default" style="font-family:'times new roman',serif">ConnectionError: [Errno 111] Connection refused. That means that the remote server was reachable, but when TCP sends the SYN packet to the server, it responds with a TCP RESET packet. So I know my network is good.</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">I decided to do some troubleshooting with tcpdump and wireshark. Chrome makes 4 connections from different source ports to port 80. I haven't a clue why, it makes the second third, and fourth connections before it ACKs the first connection. Then it does a GET /dashboard HTTP/1.1 from the first connection. The server responds with a 302 redirect to Location: <a href="http://controller1-prod.sea.opencandy.com/dashboard/admin/\r\n">http://controller1-prod.sea.opencandy.com/dashboard/admin/\r\n</a> which I guess makes sense. Then server closes the first connection. The second connection then does a GET to /dashboard/admin. The server responds with a 500 INTERNAL SERVER ERROR. The HTTP header of the response is:</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><div class="gmail_default">HTTP/1.1 500 INTERNAL SERVER ERROR</div><div class="gmail_default">Date: Thu, 31 Jul 2014 18:44:36 GMT</div><div class="gmail_default">
Server: Apache/2.2.15 (CentOS)</div><div class="gmail_default">Vary: Accept-Language,Cookie</div><div class="gmail_default">X-Frame-Options: SAMEORIGIN</div><div class="gmail_default">Content-Language: en</div><div class="gmail_default">
Set-Cookie: sessionid=".eJy1lVtz20QUx0OI48YhCRQa7rQUKA5QWfKubuF--..."; httponly; Path=/</div><div class="gmail_default">Content-Length: 1666</div><div class="gmail_default">Connection: close</div><div class="gmail_default">
Content-Type: text/html; charset=utf-8</div></div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">The error_log file contains</div>
<div class="gmail_default"><div class="gmail_default"><font face="courier new, monospace">[Thu Jul 31 00:19:59 2014] [error] File "/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send</font></div>
<div class="gmail_default"><font face="courier new, monospace">[Thu Jul 31 00:19:59 2014] [error] r = adapter.send(request, **kwargs)</font></div><div class="gmail_default"><font face="courier new, monospace">[Thu Jul 31 00:19:59 2014] [error] File "/usr/lib/python2.6/site-packages/requests/adapters.py", line 165, in send</font></div>
<div class="gmail_default"><font face="courier new, monospace">[Thu Jul 31 00:19:59 2014] [error] TypeError: function takes exactly 1 argument (3 given)</font></div><div style="font-family:'times new roman',serif">
<br></div><div style="font-family:'times new roman',serif"><br></div></div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">
Then the second session is closed down. I don't see where the third and fourth connections shut down. My hypothesis is that Chrome is opening sessions at startup to optimize performance, and that this is confusing the server. HTTP is supposed to stateless, but HTTP runs on top of TCP and TCP is stateful, so isn't clear to me if browser or the server is violating RFC 2616 (see the abstract).</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">Firefox has one connection. It does a GET /dashboard and gets a 200 response. Then the server closes the connection. Next, firefox does a request for /static/dashboard/img/logo-splash.png. The server responds with a 304 Not Modified. That makes sense - I've run this test a lot and the .png file is in the browser's cache. Firefox does a POST /dashboard./auth/login and this time, it specifies connection keep-alive. The server replies with a 200 OK (should be a 401 unauthorized code, but that's a digression) that the login failed.</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">It pains me to say this, because I used to work for Google, but I think that Google's optimization is confusing the server and causing the 500 INTERNAL SERVER ERROR.</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">As for my problems with Firefox, I tested to see if the authentication server is working.</div>
<br></div><div><div class="gmail_default"><span style="font-family:'times new roman',serif"></span><font face="courier new, monospace">root@controller1-prod.controller1-prod:/var/log/httpd# curl $OS_AUTH_URL</font></div>
<div class="gmail_default"><font face="courier new, monospace">{"version": {"status": "stable", "updated": "2012-10-13T17:42:56Z", "media-types": [{"base": "application/json", "type": "application/vnd.openstack.identity-v2.0+json"}, {"base": "application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}], "id": "v2.0", "links": [{"href": "<a href="http://controller1-prod.sea.opencandy.com:35357/v2.0/">http://controller1-prod.sea.opencandy.com:35357/v2.0/</a>", "rel": "self"}, {"href": "<a href="http://docs.openstack.org/api/openstack-identity-service/2.0/content/">http://docs.openstack.org/api/openstack-identity-service/2.0/content/</a>", "type": "text/html", "rel": "describedby"}, {"href": "<a href="http://docs.openstack.org/api/openstack-identity-service/2.root@controller1-prod.controller1-prod:/var/log/httpd#">http://docs.openstack.org/api/openstack-identity-service/2.root@controller1-prod.controller1-prod:/var/log/httpd#</a> </font></div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">I also want to see if the identity server is working, so I used keystone:</div>
<div class="gmail_default"><div class="gmail_default"><font face="courier new, monospace">root@controller1-prod.controller1-prod:/var/log/httpd# keystone role-list</font></div><div class="gmail_default"><font face="courier new, monospace">+----------------------------------+-------+</font></div>
<div class="gmail_default"><font face="courier new, monospace">| id | name |</font></div><div class="gmail_default"><font face="courier new, monospace">+----------------------------------+-------+</font></div>
<div class="gmail_default"><font face="courier new, monospace">| 601553426dc749589cc86a05a946aa47 | admin |</font></div><div class="gmail_default"><font face="courier new, monospace">+----------------------------------+-------+</font></div>
<div class="gmail_default"><font face="courier new, monospace">root@controller1-prod.controller1-prod:/var/log/httpd#</font></div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">
<br></div><div class="gmail_default" style="font-family:'times new roman',serif">I am keeping careful notes of the problems I am running into, and I hope to publish a troubleshooting/validation page for you.</div>
<div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">
Thank you for your kind assistance.</div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div><div class="gmail_default" style="font-family:'times new roman',serif">
Jeff</div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div></div><div class="gmail_default" style="font-family:'times new roman',serif"><br></div></div><br></div>-- <br><div dir="ltr">
<b>Jeff Silverman</b><div>Systems Engineer</div><div>(253) 459-2318 (c)</div><div><img src="https://dl.dropboxusercontent.com/u/16943296/SweetLabs-Signatures/New_2014/signature-logo.png"><br></div></div>
</div>