Open Stack

This is kinda hard to describe.  I am having problems with dashboard.

I am running on Centos 6.5

Dashboard is installed on my controller node.  When I first installed
dashboard and tested, I was able to login as "jeff" but then getting 500
INTERNAL SERVER ERROR responses with both Mozilla Firefox and Google
Chrome.  I looked through /var/log/error_log and found that the problem was
a ConnectionError in  /usr/lib/python2.6/site-packages/requests/adapters.py
line 206.  I never resolved that issue.  I sent a message to the mailing
list at python-requests, which is the project that write adapters.py asking
them for more troubleshooting information from the exception handler.

I rebooted the controller node to test if the services would come back.
 openstack-nova-api failed to start because:

2014-07-30 13:28:52.356 4651 ERROR nova.wsgi [-] Could not bind to 0.0.0.0:8775
2014-07-30 13:28:52.357 4651 CRITICAL nova [-] error: [Errno 98]
Address already in use

nova-api-metadata has already grabbed port 8775.  I don't understand why
nova processes are running on my controller when they are compute
processes, but then there is a lot about openstack that I don't understand
(and I appreciate your bearing with me).

​I am back to troubleshooting dashboard, and here is where it gets weird.
 It doesn't work, but it doesn't work differently depending on whether ​I
use Google Chrome (Version 36.0.1985.125 on Fedora 20) or Mozilla Firefox
(version 24.7.0-1.el6.centos).  With firefox, I get a login page.  When I
enter my username and password, I get a login failure.  error_log says [Thu
Jul 31 18:57:27 2014] [error] Login failed for user "jeff".  By way of
contrast, when I use Chrome, I get a 500 INTERNAL SERVER ERROR.  error_log
says
​ConnectionError: [Errno 111] Connection refused.  That means that the
remote server was reachable, but when TCP sends the SYN packet to the
server, it responds with a TCP RESET packet.  So I know my network is good.

I decided to do some troubleshooting with tcpdump and wireshark.  Chrome
makes 4 connections from different source ports to port 80.  I haven't a
clue why, it makes the second third, and fourth connections before it ACKs
the first connection.  Then it does a GET /dashboard HTTP/1.1 from the
first connection.  The server responds with a 302 redirect to Location:
http://controller1-prod.sea.opencandy.com/dashboard/admin/\r\n which I
guess makes sense.  Then server closes the first connection.  The second
connection then does a GET to /dashboard/admin.  The server responds with a
500 INTERNAL SERVER ERROR.  The HTTP header of the response is:
HTTP/1.1 500 INTERNAL SERVER ERROR
Date: Thu, 31 Jul 2014 18:44:36 GMT
Server: Apache/2.2.15 (CentOS)
Vary: Accept-Language,Cookie
X-Frame-Options: SAMEORIGIN
Content-Language: en
Set-Cookie: sessionid=".eJy1lVtz20QUx0OI48YhCRQa7rQUKA5QWfKubuF--...";
httponly; Path=/
Content-Length: 1666
Connection: close
Content-Type: text/html; charset=utf-8

The error_log file contains
[Thu Jul 31 00:19:59 2014] [error]   File
"/usr/lib/python2.6/site-packages/requests/sessions.py", line 374, in send
[Thu Jul 31 00:19:59 2014] [error]     r = adapter.send(request, **kwargs)
[Thu Jul 31 00:19:59 2014] [error]   File
"/usr/lib/python2.6/site-packages/requests/adapters.py", line 165, in send
[Thu Jul 31 00:19:59 2014] [error] TypeError: function takes exactly 1
argument (3 given)



​Then the second session is closed down.​  I don't see where the third and
fourth connections shut down.  My hypothesis is that Chrome is opening
sessions at startup to optimize performance, and that this is confusing the
server.  HTTP is supposed to stateless, but HTTP runs on top of TCP and TCP
is stateful, so isn't clear to me if browser or the server is violating RFC
2616 (see the abstract).

Firefox has one connection.  It does a GET /dashboard and gets a 200
response.  Then the server closes the connection.   Next, firefox does a
request for  /static/dashboard/img/logo-splash.png.  The server responds
with a 304 Not Modified.  That makes sense - I've run this test a lot and
the .png file is in the browser's cache.  Firefox does a POST
/dashboard./auth/login and this time, it specifies connection keep-alive.
 The server replies with a 200 OK (should be a 401 unauthorized code, but
that's a digression)  that the login failed.

It pains me to say this, because I used to work for Google, but I think
that Google's optimization is confusing the server and causing the 500
INTERNAL SERVER ERROR.

As for my problems with Firefox, I tested to see if the authentication
server is working.

​root at controller1-prod.controller1-prod:/var/log/httpd# curl $OS_AUTH_URL
{"version": {"status": "stable", "updated": "2012-10-13T17:42:56Z",
"media-types": [{"base": "application/json", "type":
"application/vnd.openstack.identity-v2.0+json"}, {"base":
"application/xml", "type": "application/vnd.openstack.identity-v2.0+xml"}],
"id": "v2.0", "links": [{"href": "
http://controller1-prod.sea.opencandy.com:35357/v2.0/", "rel": "self"},
{"href": "
http://docs.openstack.org/api/openstack-identity-service/2.0/content/",
"type": "text/html", "rel": "describedby"}, {"href": "
http://docs.openstack.org/api/openstack-identity-service/2.root@controller1-prod.controller1-prod:/var/log/httpd#


I also want to see if the identity server is working, so I used keystone:
root at controller1-prod.controller1-prod:/var/log/httpd# keystone role-list
+----------------------------------+-------+
|                id                |  name |
+----------------------------------+-------+
| 601553426dc749589cc86a05a946aa47 | admin |
+----------------------------------+-------+
root at controller1-prod.controller1-prod:/var/log/httpd#


I am keeping careful notes of the problems I am running into, and I hope to
publish a troubleshooting/validation page for you.


Thank you for your kind assistance.


Jeff



-- 
*Jeff Silverman*
Systems Engineer
(253) 459-2318 (c)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140731/34e77c57/attachment.html>