<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p>Hello all, <br>
</p>
<p>I'm currently having a issue whereby I'm unable to login with
horizon even after logs show that authentication was successful. I
am able to use the command line successfully without issue. In my
troubleshooting I've tried to understand how permissions and
access work in horizon to try to identify where the issue lies. To
understand how this is setup, let me first say that this isn't a
full openstack deployment. It's a "swift" stack deployment if it
can even be called that. It consists of
horizon/keystone/swift/siwft-proxy setup in a high availability
setup using haproxy and dual virtual nodes for each service with
the excption of the swift storage nodes being dedicated. There are
two HAProxy VM's with a VIP fronting the entire stack. Behind the
proxies there are two keystone nodes running keystone, openstack
dashboard. There are also two swift proxies, and 3 database nodes
in a galera cluster. SSL is enabled on keystone and horizon SSL is
terminated at the proxy as I've not net gotten around to figuring
out how to have to not need proxy termination for the SSL. Running
on Ubuntu 16.04 and Ocata release. Upgraded from newton in attempt
to fix <a class="moz-txt-link-freetext" href="https://bugs.launchpad.net/horizon/+bug/1587125">https://bugs.launchpad.net/horizon/+bug/1587125</a> when seen
that it wasn't in the install, installed patch myself to alleviate
the message below, but it wasn't truly fixed until Rob suggested
setting SESSION_ENGINE.<br>
</p>
<blockquote type="cite">Using signed cookies as SESSION_ENGINE with
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT is enabled. This disables
the ability to perform identity operations due to cookie size
constraints.</blockquote>
The exact message I get is
<pre style="margin: 0px; padding: 5px 0px; font-family: "Bitstream Vera Sans Mono", monospace; font-size: 13px; color: rgb(0, 0, 0); font-style: normal; font-variant-ligatures: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: 2; text-align: start; text-indent: 0px; text-transform: none; widows: 2; word-spacing: 0px; -webkit-text-stroke-width: 0px;">You do not have permission to access the resource:
/horizon/project/
Login as different user or go back to home page
Unauthorized. Please try logging in again.
</pre>
Openstack CLI commands work perfectly and based upon the role
assignments, the admin account has the proper permissions. I tried
granting _member_ permission to service project in an attempt to see
if that would work but it didn't. <br>
<br>
<pre>root@keystone01:/etc/keystone# openstack role assignment list --user admin --names</pre>
<pre>+----------+---------------+-------+-----------------------+---------+-----------+</pre>
<pre>| Role | User | Group | Project | Domain | Inherited |</pre>
<pre>+----------+---------------+-------+-----------------------+---------+-----------+</pre>
<pre>| _member_ | admin@Default | | service@Default | | False |</pre>
<pre>| admin | admin@Default | | service@Default | | False |</pre>
<pre>| user | admin@Default | | admin@Default | | False |</pre>
<pre>| _member_ | admin@Default | | admin@Default | | False |</pre>
<pre>| admin | admin@Default | | admin@Default | | False |</pre>
<pre>| user | admin@Default | | | Default | False |</pre>
<pre>| _member_ | admin@Default | | | Default | False |</pre>
<pre>| admin | admin@Default | | | Default | False |</pre>
<pre>+----------+---------------+-------+-----------------------+---------+-----------+</pre>
<pre>root@keystone01:/etc/keystone# openstack role assignment list --project service --names
+----------+----------------------+-------+-----------------+--------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+----------+----------------------+-------+-----------------+--------+-----------+
| admin | admin2@Default | | service@Default | | False |
| _member_ | admin@Default | | service@Default | | False |
| admin | admin@Default | | service@Default | | False |
| admin | swift@Default | | service@Default | | False |
+----------+----------------------+-------+-----------------+--------+-----------+</pre>
<pre># openstack service list</pre>
<pre>+----------------------------------+----------+--------------+</pre>
<pre>| ID | Name | Type |</pre>
<pre>+----------------------------------+----------+--------------+</pre>
<pre>| 15a143d8b27f4582823b6715b12a5fed | keystone | identity |</pre>
<pre>| d63f707495294973af260b72759f5640 | swift | object-store |</pre>
<pre>+----------------------------------+----------+--------------+</pre>
<p>I can list projects and users using the cli, I can view the
configured endpoints.<br>
</p>
<pre>root@keystone01:/etc/keystone# openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------+
| 45bce93c90a54f7c811a35624e481919 | RegionOne | swift | object-store | True | admin | <a class="moz-txt-link-freetext" href="https://">https://</a><REDACTED>:8443/v1 |
| 6004116e8eb64b5b8e2174e1a1315a7e | RegionOne | swift | object-store | True | internal | <a class="moz-txt-link-freetext" href="https://">https://</a><REDACTED>:8443/v1/AUTH_%(tenant_id)s |
| 7185955b93e94df4b6830511bd63b6c8 | RegionOne | keystone | identity | True | internal | <a class="moz-txt-link-freetext" href="https://">https://</a><REDACTED>:35357/v3/ |
| b59b879c52ac45b78e8a22ed0d4b18f3 | RegionOne | keystone | identity | True | admin | <a class="moz-txt-link-freetext" href="https://">https://</a><REDACTED>:35357/v3/ |
| bcb8690319684715a87b1144be416976 | RegionOne | swift | object-store | True | public | <a class="moz-txt-link-freetext" href="https://">https://</a><REDACTED>:8443/v1/AUTH_%(tenant_id)s |
| f036b29049e24bbd80270f6df5d491c1 | RegionOne | keystone | identity | True | public | <a class="moz-txt-link-freetext" href="https://">https://</a><REDACTED>:5000/v3/ |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------+
</pre>
<p>Here are some log entires that I see, the SESSION Engine message
posted above went away after uncommenting in config file per Rob's
suggestion.</p>
<pre><tt>[Wed Jul 05 21:17:16.530208 2017] [wsgi:error] [pid 18511:tid 140091852330752] Using signed cookies as SESSION_ENGINE with OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT is enabled. This disables the ability to perform identity operations due to cookie size constraints.
[Wed Jul 05 21:17:16.531834 2017] [wsgi:error] [pid 18511:tid 140091852330752] Login successful for user "admin", remote address <REDACTED>.</tt></pre>
I've pasted my keystone.conf here
<a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615213/">http://paste.openstack.org/show/615213/</a><br>
My openstack-dashboard/local_settings.py is here
<a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615084/">http://paste.openstack.org/show/615084/</a><br>
The horizon policy file is configured as follows and remains
untouched form when it was installed.
<a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615214/">http://paste.openstack.org/show/615214/</a>
<a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615215/">http://paste.openstack.org/show/615215/</a><br>
<br>
I chatted with Rob Cresswell in #openstack-horizon but we were
unable to come to conclusion as to cause of issue. Pasting ONLY our
conversation below so don't have to type so much as to explain our
troubleshooting in the short time we were able to chat. <br>
<pre><font face="Bitstream Vera Sans Mono,monospace">14:49 < CowboyPride> robcresswell: Aything you can point me at in terms of documentation on HOW openstack horizon manages permissions/access so I can hopefully solve my permisssions
issue. It's driving me crazy and I know it's likely something simple. I don't think it's a bug of any sort, just someting done wrong/missing on my part.
14:52 <@robcresswell> CowboyPride: Do those credentials work on the CLI?
14:53 <@robcresswell> CowboyPride: It might be because you're using the cookie backend, and the cookie is too big, so Horizon is breaking
14:53 < CowboyPride> Yes, they work on clie..
14:53 < CowboyPride> *CLI
14:54 <@robcresswell> CowboyPride: Can you please change your session backend?
14:54 < CowboyPride> Yes, I'll try that.
15:00 < CowboyPride> Didn't help, in logs for apache I only show this log entry [wsgi:error] [pid 25997:tid 140462497220352] Login successful for user "admin", remote address
15:00 < CowboyPride> hang on while I take a look at wsgi log.
15:02 < CowboyPride> I see the token request GET/POST in wsgi but not much else.
15:02 < CowboyPride> Restarted apache after setting seesion
15:03 < CowboyPride> I uncommented SESSION_ENGINE = 'django.contrib.sessions.backends.cache
15:03 < CowboyPride> That's what you ment by the session backend ^ correct?
15:03 <@robcresswell> Yep
15:04 < CowboyPride> I've even tried a apt install --reinstall openstack-dashboard to make sure in all my troubleshooting I didn't break any files in my attempt at modifying file permssison..
15:05 < CowboyPride> So all file permissions are back to their defaults.
15:06 < CowboyPride> Trying to put that in url just redirects back to permission dedined for resource but now url has changed in the error message to /horizon/project/instances/
15:06 <@robcresswell> CowboyPride: Can you carry out usual CLI operations, like listing resources (Instance, networks, etc?)
15:07 < CowboyPride> Yes, CLI works completly without issue... and curl works completly with out issue..
15:07 < CowboyPride> Only horizon is what isnt' working.
15:07 <@robcresswell> :/
15:07 <@robcresswell> So, you login successfully, but can't do anything :/
15:08 < CowboyPride> Yep and it's very strange. I did a reinstall of horizon and in order to get access back to swift content post install I had to updte the ID string for admin user in the user table, assignments table etc and that is when horizon broke.
15:09 < CowboyPride> Provided the ID has been updated everywhere it should work.
15:09 <@robcresswell> :/ That sounds more like a Keystone issue
15:09 < CowboyPride> and the cli is a testment that the id replace worked.
15:09 <@robcresswell> Yeah
15:11 < CowboyPride> Here is my keystone file <a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615213/">http://paste.openstack.org/show/615213/</a>
15:11 <@robcresswell> CowboyPride: It could be a policy issue with v3. But at this point I'm not doing much more than slightly-educated-guesses
15:12 < CowboyPride> Not using Nova/Glance/Neutron etc... THis is purely a "Swift" stack if it can be called that.
15:12 <@robcresswell> OH
15:12 < CowboyPride> Only thing I have setup is Keystone/Horizon/Swift/Swift-Proxy
15:12 <@robcresswell> Now I have an idea
15:13 <@robcresswell> Although, I'd have thought this would fail before, so maybe not...
15:13 <@robcresswell> CowboyPride: When you reinstalled, did you change the enabled/ files to reference Swift as the default panel?
15:14 <@robcresswell> It might be logging in, attempting to load the default view (Overview? iirc) failing all the permissions checks, and not knowing what to do.
15:14 < CowboyPride> No, the only thing I did was reinstall and then test that eveyrthing worked, restored users accounts, swift endpoint, updated admin user id.
15:15 < CowboyPride> After the admin userid update to be able to view the swift content again that when it broke.
15:15 <@robcresswell> But it worked find before that?
15:16 < CowboyPride> Yes, I was able to view all the accounts I restored into horizon, I was able to view permissions, etch.
15:16 <@robcresswell> Hmm, I'm stumped then
15:16 < CowboyPride> I tired changing the id back to the newly reinstally admin user id thinking it as refrencing it somewhere but no such luck.
15:17 < CowboyPride> kept wondering if there is a hidden refrence it the boostrapping that refrences the id somewhere and changing it broke.
15:17 < CowboyPride> but when I canged it back and it didn't give me access back in horizon I abandonded that thought.
15:18 < CowboyPride> *changed
15:20 < CowboyPride> You're not the only one. Everying I'm looking says it should be working just fine.
15:21 <@robcresswell> CowboyPride: Can you paste your copy of the keystone policy file in Horizon?
15:21 <@robcresswell> Wanted to see something
15:22 < CowboyPride> sure
15:23 < CowboyPride> keystone_policy.json under /usr/share/openstack-dashboard/openstack_dashboard?
15:24 < CowboyPride> <a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615214/">http://paste.openstack.org/show/615214/</a>
15:25 < CowboyPride> robcresswell: There it is ^
15:26 < CowboyPride> And here is /etc/keystone/policy.json <a class="moz-txt-link-freetext" href="http://paste.openstack.org/show/615215/">http://paste.openstack.org/show/615215/</a>
15:28 <@robcresswell> CowboyPride: Huh, they're different
15:28 < CowboyPride> Yep, I and tried copying the /keystone/policy.json over to the openstack one and it didn't help either...
15:28 <@robcresswell> :/
15:28 < CowboyPride> reverted the change after it didnt' work.
15:28 <@robcresswell> I'm all out of sleepy ideas
15:29 <@robcresswell> Try an email on the dev mailer
15:29 <@robcresswell> Or asking in #openstack-keystone.
15:29 <@robcresswell> They may have some suggestions
15:29 <@robcresswell> Mailing list probably best though, bigger audience and easier to follow async.
15:29 <@robcresswell> Tag horizon and keystone in the subject :)
15:30 < CowboyPride> Okay. Thanks, I'll do that.
15:30 * robcresswell heads to bed</font>
</pre>
<br>
One final troubleshooting idea I tired was creating myself a user
account with admin permissions and it gets the same message too, but
the openstack command set works with my new personal admin account.
<pre>root@keystone01:/etc/keystone# openstack role assignment list --user caseyrichins --names</pre>
<pre>+----------+----------------------+-------+-----------------+---------+-----------+</pre>
<pre>| Role | User | Group | Project | Domain | Inherited |</pre>
<pre>+----------+----------------------+-------+-----------------+---------+-----------+</pre>
<pre>| admin | caseyrichins@Default | | service@Default | | False |</pre>
<pre>| user | caseyrichins@Default | | admin@Default | | False |</pre>
<pre>| _member_ | caseyrichins@Default | | admin@Default | | False |</pre>
<pre>| admin | caseyrichins@Default | | admin@Default | | False |</pre>
<pre>| user | caseyrichins@Default | | | Default | False |</pre>
<pre>| _member_ | caseyrichins@Default | | | Default | False |</pre>
<pre>| admin | caseyrichins@Default | | | Default | False |</pre>
<pre>+----------+----------------------+-------+-----------------+---------+-----------+</pre>
Thanks in advance to any assistance anyone is able to provide.<br>
<br>
Regards,<br>
<br>
Casey C. Richins<br>
Linux System Administrator<br>
Fibernet Corp. <br>
</body>
</html>