[Openstack] [keystone] [horizon] - Resource Permission Denied error on horizon login after successful authentication
Casey Richins
caseyrichins at fiber.net
Wed Jul 12 22:38:20 UTC 2017
Hello all,
I'm currently having a issue whereby I'm unable to login with horizon
even after logs show that authentication was successful. I am able to
use the command line successfully without issue. In my troubleshooting
I've tried to understand how permissions and access work in horizon to
try to identify where the issue lies. To understand how this is setup,
let me first say that this isn't a full openstack deployment. It's a
"swift" stack deployment if it can even be called that. It consists of
horizon/keystone/swift/siwft-proxy setup in a high availability setup
using haproxy and dual virtual nodes for each service with the excption
of the swift storage nodes being dedicated. There are two HAProxy VM's
with a VIP fronting the entire stack. Behind the proxies there are two
keystone nodes running keystone, openstack dashboard. There are also two
swift proxies, and 3 database nodes in a galera cluster. SSL is enabled
on keystone and horizon SSL is terminated at the proxy as I've not net
gotten around to figuring out how to have to not need proxy termination
for the SSL. Running on Ubuntu 16.04 and Ocata release. Upgraded from
newton in attempt to fix https://bugs.launchpad.net/horizon/+bug/1587125
when seen that it wasn't in the install, installed patch myself to
alleviate the message below, but it wasn't truly fixed until Rob
suggested setting SESSION_ENGINE.
> Using signed cookies as SESSION_ENGINE with
> OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT is enabled. This disables the
> ability to perform identity operations due to cookie size constraints.
The exact message I get is
You do not have permission to access the resource:
/horizon/project/
Login as different user or go back to home page
Unauthorized. Please try logging in again.
Openstack CLI commands work perfectly and based upon the role
assignments, the admin account has the proper permissions. I tried
granting _member_ permission to service project in an attempt to see if
that would work but it didn't.
root at keystone01:/etc/keystone# openstack role assignment list --user admin --names
+----------+---------------+-------+-----------------------+---------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+----------+---------------+-------+-----------------------+---------+-----------+
| _member_ | admin at Default | | service at Default | | False |
| admin | admin at Default | | service at Default | | False |
| user | admin at Default | | admin at Default | | False |
| _member_ | admin at Default | | admin at Default | | False |
| admin | admin at Default | | admin at Default | | False |
| user | admin at Default | | | Default | False |
| _member_ | admin at Default | | | Default | False |
| admin | admin at Default | | | Default | False |
+----------+---------------+-------+-----------------------+---------+-----------+
root at keystone01:/etc/keystone# openstack role assignment list --project service --names
+----------+----------------------+-------+-----------------+--------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+----------+----------------------+-------+-----------------+--------+-----------+
| admin | admin2 at Default | | service at Default | | False |
| _member_ | admin at Default | | service at Default | | False |
| admin | admin at Default | | service at Default | | False |
| admin | swift at Default | | service at Default | | False |
+----------+----------------------+-------+-----------------+--------+-----------+
# openstack service list
+----------------------------------+----------+--------------+
| ID | Name | Type |
+----------------------------------+----------+--------------+
| 15a143d8b27f4582823b6715b12a5fed | keystone | identity |
| d63f707495294973af260b72759f5640 | swift | object-store |
+----------------------------------+----------+--------------+
I can list projects and users using the cli, I can view the configured
endpoints.
root at keystone01:/etc/keystone# openstack endpoint list
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------+
| ID | Region | Service Name | Service Type | Enabled | Interface | URL |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------+
| 45bce93c90a54f7c811a35624e481919 | RegionOne | swift | object-store | True | admin | https://<REDACTED>:8443/v1 |
| 6004116e8eb64b5b8e2174e1a1315a7e | RegionOne | swift | object-store | True | internal | https://<REDACTED>:8443/v1/AUTH_%(tenant_id)s |
| 7185955b93e94df4b6830511bd63b6c8 | RegionOne | keystone | identity | True | internal | https://<REDACTED>:35357/v3/ |
| b59b879c52ac45b78e8a22ed0d4b18f3 | RegionOne | keystone | identity | True | admin | https://<REDACTED>:35357/v3/ |
| bcb8690319684715a87b1144be416976 | RegionOne | swift | object-store | True | public | https://<REDACTED>:8443/v1/AUTH_%(tenant_id)s |
| f036b29049e24bbd80270f6df5d491c1 | RegionOne | keystone | identity | True | public | https://<REDACTED>:5000/v3/ |
+----------------------------------+-----------+--------------+--------------+---------+-----------+-----------------------------------------------------+
Here are some log entires that I see, the SESSION Engine message posted
above went away after uncommenting in config file per Rob's suggestion.
[Wed Jul 05 21:17:16.530208 2017] [wsgi:error] [pid 18511:tid
140091852330752] Using signed cookies as SESSION_ENGINE with
OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT is enabled. This disables the
ability to perform identity operations due to cookie size constraints.
[Wed Jul 05 21:17:16.531834 2017] [wsgi:error] [pid 18511:tid
140091852330752] Login successful for user "admin", remote address
<REDACTED>.
I've pasted my keystone.conf here http://paste.openstack.org/show/615213/
My openstack-dashboard/local_settings.py is here
http://paste.openstack.org/show/615084/
The horizon policy file is configured as follows and remains untouched
form when it was installed. http://paste.openstack.org/show/615214/
http://paste.openstack.org/show/615215/
I chatted with Rob Cresswell in #openstack-horizon but we were unable to
come to conclusion as to cause of issue. Pasting ONLY our conversation
below so don't have to type so much as to explain our troubleshooting in
the short time we were able to chat.
14:49 < CowboyPride> robcresswell: Aything you can point me at in terms
of documentation on HOW openstack horizon manages permissions/access so
I can hopefully solve my permisssions issue. It's driving me crazy and I
know it's likely something simple. I don't think it's a bug of any sort,
just someting done wrong/missing on my part. 14:52 <@robcresswell>
CowboyPride: Do those credentials work on the CLI? 14:53 <@robcresswell>
CowboyPride: It might be because you're using the cookie backend, and
the cookie is too big, so Horizon is breaking 14:53 < CowboyPride> Yes,
they work on clie.. 14:53 < CowboyPride> *CLI 14:54 <@robcresswell>
CowboyPride: Can you please change your session backend? 14:54 <
CowboyPride> Yes, I'll try that. 15:00 < CowboyPride> Didn't help, in
logs for apache I only show this log entry [wsgi:error] [pid 25997:tid
140462497220352] Login successful for user "admin", remote address 15:00
< CowboyPride> hang on while I take a look at wsgi log. 15:02 <
CowboyPride> I see the token request GET/POST in wsgi but not much else.
15:02 < CowboyPride> Restarted apache after setting seesion 15:03 <
CowboyPride> I uncommented SESSION_ENGINE =
'django.contrib.sessions.backends.cache 15:03 < CowboyPride> That's what
you ment by the session backend ^ correct? 15:03 <@robcresswell> Yep
15:04 < CowboyPride> I've even tried a apt install --reinstall
openstack-dashboard to make sure in all my troubleshooting I didn't
break any files in my attempt at modifying file permssison.. 15:05 <
CowboyPride> So all file permissions are back to their defaults. 15:06 <
CowboyPride> Trying to put that in url just redirects back to permission
dedined for resource but now url has changed in the error message to
/horizon/project/instances/ 15:06 <@robcresswell> CowboyPride: Can you
carry out usual CLI operations, like listing resources (Instance,
networks, etc?) 15:07 < CowboyPride> Yes, CLI works completly without
issue... and curl works completly with out issue.. 15:07 < CowboyPride>
Only horizon is what isnt' working. 15:07 <@robcresswell> :/ 15:07
<@robcresswell> So, you login successfully, but can't do anything :/
15:08 < CowboyPride> Yep and it's very strange. I did a reinstall of
horizon and in order to get access back to swift content post install I
had to updte the ID string for admin user in the user table, assignments
table etc and that is when horizon broke. 15:09 < CowboyPride> Provided
the ID has been updated everywhere it should work. 15:09 <@robcresswell>
:/ That sounds more like a Keystone issue 15:09 < CowboyPride> and the
cli is a testment that the id replace worked. 15:09 <@robcresswell> Yeah
15:11 < CowboyPride> Here is my keystone file
http://paste.openstack.org/show/615213/ 15:11 <@robcresswell>
CowboyPride: It could be a policy issue with v3. But at this point I'm
not doing much more than slightly-educated-guesses 15:12 < CowboyPride>
Not using Nova/Glance/Neutron etc... THis is purely a "Swift" stack if
it can be called that. 15:12 <@robcresswell> OH 15:12 < CowboyPride>
Only thing I have setup is Keystone/Horizon/Swift/Swift-Proxy 15:12
<@robcresswell> Now I have an idea 15:13 <@robcresswell> Although, I'd
have thought this would fail before, so maybe not... 15:13
<@robcresswell> CowboyPride: When you reinstalled, did you change the
enabled/ files to reference Swift as the default panel? 15:14
<@robcresswell> It might be logging in, attempting to load the default
view (Overview? iirc) failing all the permissions checks, and not
knowing what to do. 15:14 < CowboyPride> No, the only thing I did was
reinstall and then test that eveyrthing worked, restored users accounts,
swift endpoint, updated admin user id. 15:15 < CowboyPride> After the
admin userid update to be able to view the swift content again that when
it broke. 15:15 <@robcresswell> But it worked find before that? 15:16 <
CowboyPride> Yes, I was able to view all the accounts I restored into
horizon, I was able to view permissions, etch. 15:16 <@robcresswell>
Hmm, I'm stumped then 15:16 < CowboyPride> I tired changing the id back
to the newly reinstally admin user id thinking it as refrencing it
somewhere but no such luck. 15:17 < CowboyPride> kept wondering if there
is a hidden refrence it the boostrapping that refrences the id somewhere
and changing it broke. 15:17 < CowboyPride> but when I canged it back
and it didn't give me access back in horizon I abandonded that thought.
15:18 < CowboyPride> *changed 15:20 < CowboyPride> You're not the only
one. Everying I'm looking says it should be working just fine. 15:21
<@robcresswell> CowboyPride: Can you paste your copy of the keystone
policy file in Horizon? 15:21 <@robcresswell> Wanted to see something
15:22 < CowboyPride> sure 15:23 < CowboyPride> keystone_policy.json
under /usr/share/openstack-dashboard/openstack_dashboard? 15:24 <
CowboyPride> http://paste.openstack.org/show/615214/ 15:25 <
CowboyPride> robcresswell: There it is ^ 15:26 < CowboyPride> And here
is /etc/keystone/policy.json http://paste.openstack.org/show/615215/
15:28 <@robcresswell> CowboyPride: Huh, they're different 15:28 <
CowboyPride> Yep, I and tried copying the /keystone/policy.json over to
the openstack one and it didn't help either... 15:28 <@robcresswell> :/
15:28 < CowboyPride> reverted the change after it didnt' work. 15:28
<@robcresswell> I'm all out of sleepy ideas 15:29 <@robcresswell> Try an
email on the dev mailer 15:29 <@robcresswell> Or asking in
#openstack-keystone. 15:29 <@robcresswell> They may have some
suggestions 15:29 <@robcresswell> Mailing list probably best though,
bigger audience and easier to follow async. 15:29 <@robcresswell> Tag
horizon and keystone in the subject :) 15:30 < CowboyPride> Okay.
Thanks, I'll do that. 15:30 * robcresswell heads to bed
One final troubleshooting idea I tired was creating myself a user
account with admin permissions and it gets the same message too, but the
openstack command set works with my new personal admin account.
root at keystone01:/etc/keystone# openstack role assignment list --user caseyrichins --names
+----------+----------------------+-------+-----------------+---------+-----------+
| Role | User | Group | Project | Domain | Inherited |
+----------+----------------------+-------+-----------------+---------+-----------+
| admin | caseyrichins at Default | | service at Default | | False |
| user | caseyrichins at Default | | admin at Default | | False |
| _member_ | caseyrichins at Default | | admin at Default | | False |
| admin | caseyrichins at Default | | admin at Default | | False |
| user | caseyrichins at Default | | | Default | False |
| _member_ | caseyrichins at Default | | | Default | False |
| admin | caseyrichins at Default | | | Default | False |
+----------+----------------------+-------+-----------------+---------+-----------+
Thanks in advance to any assistance anyone is able to provide.
Regards,
Casey C. Richins
Linux System Administrator
Fibernet Corp.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20170712/a2046e68/attachment.html>
More information about the Openstack
mailing list