[ops] Openstack Victoria SPICE configuration not working
I have posted this on stackexchange, but am also asking here:
Running openstack on Ubuntu 20.04 with a controller node and four compute nodes. I am reading the instructions on configuring SPICE here. I will admit, I found out about which packages to install from other sites, as the instructions at the link do not actually list any software that needs to be installed.
On the compute nodes and controller, I installed nova-spiceproxy On the controller I installed nova-spiceproxy and spice-html5
I followed the instructions in the link on configuring /etc/nova/nova.conf on the controller and all compute nodes.
In a nutshell, when you click on console in the dashboard, it simply says:
Something went wrong!
An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.
I have parsed every log in /var/log on the compute and controller nodes and found nothing that would indicate it is failing.
On the compute node, I can do a ps aux and see that spice is running on the instance I am trying to connect to.
I am really not sure where to go. I have tried VNC and noVNC also to no avail, each time completely removing the old configs and only adding the new. I have reverted back to SPICE to try the "simple" approach.
My relevant config items in /etc/nova/nova.conf from a compute node are:
[DEFAULT] vnc_eanabled = false # yes, I know it is redundant, I am following the docs...
[vnc] enabled = false
[spice] enabled = true agent_enabled = true html5proxy_base_url = http://10.131.39.40:6082/spice_auto.html # the above IP is the IP of the controller. server_listen = 0.0.0.0 server_proxyclient_address = 10.131.29.42 # the above IP is the IP of the controller that I pulled this config from html5proxy_host = 0.0.0.0 html5proxy_port = 6082
Back on the controller node, in /etc/nova/nova.conf I have:
[spice] enabled = true agent_enabled = false html5proxy_host = 0.0.0.0 html5proxy_port = 6082 keymap = en_us
I also have vnc false, as it is on the compute nodes. I will admit further, I have tried at least 20 iterations of this, just leaving off proxy ports, or thinking for a moment that the proxyclientaddress was the controller node, but none of them work. I do restart nova on the controller and all compute nodes after every change.
Any ideas or directions would be appreciated. I wish there were logs, I am perhaps missing them or need to turn on debugging of some kind.
On 8/5/21 7:25 PM, number9 wrote:
I have posted this on stackexchange, but am also asking here:
Running openstack on Ubuntu 20.04 with a controller node and four compute nodes. I am reading the instructions on configuring SPICE here. I will admit, I found out about which packages to install from other sites, as the instructions at the link do not actually list any software that needs to be installed.
On the compute nodes and controller, I installed nova-spiceproxy On the controller I installed nova-spiceproxy and spice-html5
I followed the instructions in the link on configuring /etc/nova/nova.conf on the controller and all compute nodes.
In a nutshell, when you click on console in the dashboard, it simply says:
Something went wrong!
An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.
I have parsed every log in /var/log on the compute and controller nodes and found nothing that would indicate it is failing.
On the compute node, I can do a ps aux and see that spice is running on the instance I am trying to connect to.
I am really not sure where to go. I have tried VNC and noVNC also to no avail, each time completely removing the old configs and only adding the new. I have reverted back to SPICE to try the "simple" approach.
My relevant config items in /etc/nova/nova.conf from a compute node are:
[DEFAULT] vnc_eanabled = false # yes, I know it is redundant, I am following the docs...
[vnc] enabled = false
[spice] enabled = true agent_enabled = true html5proxy_base_url = http://10.131.39.40:6082/spice_auto.html # the above IP is the IP of the controller. server_listen = 0.0.0.0 server_proxyclient_address = 10.131.29.42 # the above IP is the IP of the controller that I pulled this config from html5proxy_host = 0.0.0.0 html5proxy_port = 6082
Back on the controller node, in /etc/nova/nova.conf I have:
[spice] enabled = true agent_enabled = false html5proxy_host = 0.0.0.0 html5proxy_port = 6082 keymap = en_us
I also have vnc false, as it is on the compute nodes. I will admit further, I have tried at least 20 iterations of this, just leaving off proxy ports, or thinking for a moment that the proxyclientaddress was the controller node, but none of them work. I do restart nova on the controller and all compute nodes after every change.
Any ideas or directions would be appreciated. I wish there were logs, I am perhaps missing them or need to turn on debugging of some kind.
Hi,
The Debian packages have spice by default. Did you try it?
Cheers,
Thomas Goirand (zigo)
I am not sure I follow you. I am on ubuntu... do I need to install some debian spice package to get this working?
On 2021-08-06 12:11, Thomas Goirand wrote:
On 8/5/21 7:25 PM, number9 wrote:
I have posted this on stackexchange, but am also asking here:
Running openstack on Ubuntu 20.04 with a controller node and four compute nodes. I am reading the instructions on configuring SPICE here. I will admit, I found out about which packages to install from other sites, as the instructions at the link do not actually list any software that needs to be installed.
On the compute nodes and controller, I installed nova-spiceproxy On the controller I installed nova-spiceproxy and spice-html5
I followed the instructions in the link on configuring /etc/nova/nova.conf on the controller and all compute nodes.
In a nutshell, when you click on console in the dashboard, it simply says:
Something went wrong!
An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.
I have parsed every log in /var/log on the compute and controller nodes and found nothing that would indicate it is failing.
On the compute node, I can do a ps aux and see that spice is running on the instance I am trying to connect to.
I am really not sure where to go. I have tried VNC and noVNC also to no avail, each time completely removing the old configs and only adding the new. I have reverted back to SPICE to try the "simple" approach.
My relevant config items in /etc/nova/nova.conf from a compute node are:
[DEFAULT] vnc_eanabled = false # yes, I know it is redundant, I am following the docs...
[vnc] enabled = false
[spice] enabled = true agent_enabled = true html5proxy_base_url = http://10.131.39.40:6082/spice_auto.html # the above IP is the IP of the controller. server_listen = 0.0.0.0 server_proxyclient_address = 10.131.29.42 # the above IP is the IP of the controller that I pulled this config from html5proxy_host = 0.0.0.0 html5proxy_port = 6082
Back on the controller node, in /etc/nova/nova.conf I have:
[spice] enabled = true agent_enabled = false html5proxy_host = 0.0.0.0 html5proxy_port = 6082 keymap = en_us
I also have vnc false, as it is on the compute nodes. I will admit further, I have tried at least 20 iterations of this, just leaving off proxy ports, or thinking for a moment that the proxyclientaddress was the controller node, but none of them work. I do restart nova on the controller and all compute nodes after every change.
Any ideas or directions would be appreciated. I wish there were logs, I am perhaps missing them or need to turn on debugging of some kind.
Hi,
The Debian packages have spice by default. Did you try it?
Cheers,
Thomas Goirand (zigo)
I
Are you suggesting that I rebuild my six servers with Debian in order to get spice working?
I thought Ubuntu server was a supported platform (e.g. the docs say RHEL, SuSe, Ubuntu)?
Would moving to Wallaby help? I am hesitant, as I have tried to get vnc and novnc working on Rocky and Pike with the same results. The problem is I really _need_ it to work for next semester so the students can access their vms over the web for some remote classes.
Thanks.
On 2021-08-07 10:01, Thomas Goirand wrote:
On 8/6/21 11:32 PM, number9 wrote:
I am not sure I follow you. I am on ubuntu... do I need to install some debian spice package to get this working?
Probably you don't *need* it, but using Debian is another option...
Thomas
On Sat, Aug 07, 2021 at 10:10:20AM -0500, number9 wrote:
Are you suggesting that I rebuild my six servers with Debian in order to get spice working?
I thought Ubuntu server was a supported platform (e.g. the docs say RHEL, SuSe, Ubuntu)?
Would moving to Wallaby help? I am hesitant, as I have tried to get vnc and novnc working on Rocky and Pike with the same results. The problem is I really _need_ it to work for next semester so the students can access their vms over the web for some remote classes.
Thanks.
Hi,
from the error description, I can only suspect you may be able to see the error message in Horizon logs. Usually a "Something went wrong" message is the nicer way for a 500 server error.
If I understood correctly what Thomas was trying to suggest is: Ubuntu and Debian packages should work on both systems, i.e. you should be able to use the Debian packages on Ubuntu. Thomas puts a lot of effort in these.
OpenStack is quite easy to misconfigure, and you can waste some time in finding your own typos (or the one in guides). This is why installers are popular.
Matthias
Thanks for the information, I have a few questions:
What packages are you referring to? Did I install the wrong spice packages to make spice run?
You note installers are popular, but I was under the impression the only installer was the "single machine" developer installer, and not one for a controller node, five compute nodes and a block storage node. Am I missing something? Is there an installer where I can just install Openstack "controller" on the controller, "compute" on the compute nodes and say "swift" or "cinder" on my storage node, and it will all be configured to work together? That would save me some real configuration time. I have already installed it three times by hand (Rocky, Pike, Victoria), and that is a fair amount of work.
Thanks.
On 2021-08-09 02:21, Matthias Runge wrote:
On Sat, Aug 07, 2021 at 10:10:20AM -0500, number9 wrote:
Are you suggesting that I rebuild my six servers with Debian in order to get spice working?
I thought Ubuntu server was a supported platform (e.g. the docs say RHEL, SuSe, Ubuntu)?
Would moving to Wallaby help? I am hesitant, as I have tried to get vnc and novnc working on Rocky and Pike with the same results. The problem is I really _need_ it to work for next semester so the students can access their vms over the web for some remote classes.
Thanks.
Hi,
from the error description, I can only suspect you may be able to see the error message in Horizon logs. Usually a "Something went wrong" message is the nicer way for a 500 server error.
If I understood correctly what Thomas was trying to suggest is: Ubuntu and Debian packages should work on both systems, i.e. you should be able to use the Debian packages on Ubuntu. Thomas puts a lot of effort in these.
OpenStack is quite easy to misconfigure, and you can waste some time in finding your own typos (or the one in guides). This is why installers are popular.
Matthias
On Mon, Aug 09, 2021 at 05:33:18AM -0500, number9 wrote:
Thanks for the information, I have a few questions:
What packages are you referring to? Did I install the wrong spice packages to make spice run?
You note installers are popular, but I was under the impression the only installer was the "single machine" developer installer, and not one for a controller node, five compute nodes and a block storage node. Am I missing something? Is there an installer where I can just install Openstack "controller" on the controller, "compute" on the compute nodes and say "swift" or "cinder" on my storage node, and it will all be configured to work together? That would save me some real configuration time. I have already installed it three times by hand (Rocky, Pike, Victoria), and that is a fair amount of work.
I was referring to .deb packages. I don't know what you did install (and what not). Also I am not using Debian (or Ubuntu).
Installers would deploy OpenStack on your bare metal machines, and would make "things work" together.
I'd just name kolla or tripleo here, but there are more, see[1].
Single node deployments are just a special case, but are not really supportable (nor would they create a highly available infrastructure).
Matthias
[1] https://www.openstack.org/software/project-navigator/deployment-tools
Thanks.
On 2021-08-09 02:21, Matthias Runge wrote:
On Sat, Aug 07, 2021 at 10:10:20AM -0500, number9 wrote:
Are you suggesting that I rebuild my six servers with Debian in order to get spice working?
I thought Ubuntu server was a supported platform (e.g. the docs say RHEL, SuSe, Ubuntu)?
Would moving to Wallaby help? I am hesitant, as I have tried to get vnc and novnc working on Rocky and Pike with the same results. The problem is I really _need_ it to work for next semester so the students can access their vms over the web for some remote classes.
Thanks.
Hi,
from the error description, I can only suspect you may be able to see the error message in Horizon logs. Usually a "Something went wrong" message is the nicer way for a 500 server error.
If I understood correctly what Thomas was trying to suggest is: Ubuntu and Debian packages should work on both systems, i.e. you should be able to use the Debian packages on Ubuntu. Thomas puts a lot of effort in these.
OpenStack is quite easy to misconfigure, and you can waste some time in finding your own typos (or the one in guides). This is why installers are popular.
Matthias
On Thu, 2021-08-05 at 12:25 -0500, number9 wrote:
I have posted this on stackexchange, but am also asking here:
Running openstack on Ubuntu 20.04 with a controller node and four compute nodes. I am reading the instructions on configuring SPICE here. I will admit, I found out about which packages to install from other sites, as the instructions at the link do not actually list any software that needs to be installed.
On the compute nodes and controller, I installed nova-spiceproxy On the controller I installed nova-spiceproxy and spice-html5
Not an answer, but as an aside SPICE support is poorly maintained in nova and its usage not generally advised. It may be removed in a future release. I would suggest relying on noVNC instead. With that said...
I followed the instructions in the link on configuring /etc/nova/nova.conf on the controller and all compute nodes.
In a nutshell, when you click on console in the dashboard, it simply says:
Something went wrong!
An unexpected error has occurred. Try refreshing the page. If that doesn't help, contact your local administrator.
I have parsed every log in /var/log on the compute and controller nodes and found nothing that would indicate it is failing.
On the compute node, I can do a ps aux and see that spice is running on the instance I am trying to connect to.
I am really not sure where to go. I have tried VNC and noVNC also to no avail, each time completely removing the old configs and only adding the new. I have reverted back to SPICE to try the "simple" approach.
My relevant config items in /etc/nova/nova.conf from a compute node are:
[DEFAULT] vnc_eanabled = false
There's a typo here, but it shouldn't matter - that config option has been replaced by the '[vnc] enabled' option below. The doc clearly needs an update. As another aside, because virtually everyone seems to use an installer, the actual install guides get very TLC and are subsequently prone to bitrot :( However, the remote console doc at [1] was updated in recent releases and should be relatively accurate.
[1] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
# yes, I know it is redundant, I am following the docs...
[vnc] enabled = false
[spice] enabled = true agent_enabled = true html5proxy_base_url = http://10.131.39.40:6082/spice_auto.html # the above IP is the IP of the controller. server_listen = 0.0.0.0 server_proxyclient_address = 10.131.29.42
So I assume '10.131.39.40' is address of the controller and '10.131.29.42' is the address of the compute node, yes? Is the former publicly accessible from the same place you're browsing the web UX? I would assume not, given it's a private network IP. If not, this needs to be updated. This is the value returned from the API which is in turn used by Horizon iirc.
# the above IP is the IP of the controller that I pulled this config from html5proxy_host = 0.0.0.0 html5proxy_port = 6082
These are redundant (they match the defaults) but also no harm so...
Back on the controller node, in /etc/nova/nova.conf I have:
[spice] enabled = true agent_enabled = false html5proxy_host = 0.0.0.0 html5proxy_port = 6082 keymap = en_us
This all looks correct to me. Somewhat unrelated: I'm not certain whether this is true for SPICE, but you should not configure the keymap option for VNC. It's the source of bugs since the QEMU-based keymaps this enables are incomplete and buggy. Better to use a newer version of noVNC that supports native keymapping.
I also have vnc false, as it is on the compute nodes. I will admit further, I have tried at least 20 iterations of this, just leaving off proxy ports, or thinking for a moment that the proxyclientaddress was the controller node, but none of them work. I do restart nova on the controller and all compute nodes after every change.
Any ideas or directions would be appreciated. I wish there were logs, I am perhaps missing them or need to turn on debugging of some kind.
I suspect the issue is with the address used for '[spice] html5proxy_base_url'. I would suggest using the 'openstack console url show' command to get a URL to access the console without needing to invoke Horizon. If you're able to access this URL then the issue is instead somewhere in Horizon. If not, it's definitely nova. You can also enable debug-level logging for the nova-spicehtml5proxy and nova-novncproxy services to ensure you're actually seeing a connection from your browser.
Hopefully this helps somewhat, Stephen
On 2021-08-09 05:59, Stephen Finucane wrote:
On Thu, 2021-08-05 at 12:25 -0500, number9 wrote:
I have posted this on stackexchange, but am also asking here:
Running openstack on Ubuntu 20.04 with a controller node and four compute nodes. I am reading the instructions on configuring SPICE here. I will admit, I found out about which packages to install from other sites, as the instructions at the link do not actually list any software that needs to be installed.
On the compute nodes and controller, I installed nova-spiceproxy On the controller I installed nova-spiceproxy and spice-html5Not an answer, but as an aside SPICE support is poorly maintained in nova and its usage not generally advised. It may be removed in a future release. I would suggest relying on noVNC instead. With that said...
Hrm, I might try to go back to noVNC. I could never get that working either.
I followed the instructions in the link on configuring /etc/nova/nova.conf on the controller and all compute nodes.
There's a typo here, but it shouldn't matter - that config option has been replaced by the '[vnc] enabled' option below. The doc clearly needs an update. As another aside, because virtually everyone seems to use an installer, the actual install guides get very TLC and are subsequently prone to bitrot :( However, the remote console doc at [1] was updated in recent releases and should be relatively accurate.
[1] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
I have been referencing the above doc. I am just noting that I was putting in redundancies as I never know what is really working in the configs when there are differences from say one version to the next.
# yes, I know it is redundant, I am following the docs...
[vnc] enabled = false
[spice] enabled = true agent_enabled = true html5proxy_base_url = http://10.131.39.40:6082/spice_auto.html # the above IP is the IP of the controller. server_listen = 0.0.0.0 server_proxyclient_address = 10.131.29.42
So I assume '10.131.39.40' is address of the controller and '10.131.29.42' is the address of the compute node, yes? Is the former publicly accessible from the same place you're browsing the web UX? I would assume not, given it's a private network IP. If not, this needs to be updated. This is the value returned from the API which is in turn used by Horizon iirc.
YES, the private internal IIP address scheme is what people hit horizon dashboard on. The clients are am on a private internal network.
I suspect the issue is with the address used for '[spice] html5proxy_base_url'. I would suggest using the 'openstack console url show' command to get a URL to access the console without needing to invoke Horizon. If you're able to access this URL then the issue is instead somewhere in Horizon. If not, it's definitely nova. You can also enable debug-level logging for the nova-spicehtml5proxy and nova-novncproxy services to ensure you're actually seeing a connection from your browser.
I will check openstack console url show
It says I need a <server>, but no matter what I put in, I get "no server name or ID of that exists".
Huh, perhaps I am loosing my mind? I can login to horizon and spin up VMs, but this is blank:
openstack server list
As I was trying different server names on openstack console url show.
On Mon, 2021-08-09 at 10:36 -0500, number9 wrote:
On 2021-08-09 05:59, Stephen Finucane wrote:
On Thu, 2021-08-05 at 12:25 -0500, number9 wrote:
I have posted this on stackexchange, but am also asking here:
Running openstack on Ubuntu 20.04 with a controller node and four compute nodes. I am reading the instructions on configuring SPICE here. I will admit, I found out about which packages to install from other sites, as the instructions at the link do not actually list any software that needs to be installed.
On the compute nodes and controller, I installed nova-spiceproxy On the controller I installed nova-spiceproxy and spice-html5Not an answer, but as an aside SPICE support is poorly maintained in nova and its usage not generally advised. It may be removed in a future release. I would suggest relying on noVNC instead. With that said...
Hrm, I might try to go back to noVNC. I could never get that working either.
I followed the instructions in the link on configuring /etc/nova/nova.conf on the controller and all compute nodes.
There's a typo here, but it shouldn't matter - that config option has been replaced by the '[vnc] enabled' option below. The doc clearly needs an update. As another aside, because virtually everyone seems to use an installer, the actual install guides get very TLC and are subsequently prone to bitrot :( However, the remote console doc at [1] was updated in recent releases and should be relatively accurate.
[1] https://docs.openstack.org/nova/latest/admin/remote-console-access.html
I have been referencing the above doc. I am just noting that I was putting in redundancies as I never know what is really working in the configs when there are differences from say one version to the next.
# yes, I know it is redundant, I am following the docs...
[vnc] enabled = false
[spice] enabled = true agent_enabled = true html5proxy_base_url = http://10.131.39.40:6082/spice_auto.html # the above IP is the IP of the controller. server_listen = 0.0.0.0 server_proxyclient_address = 10.131.29.42
So I assume '10.131.39.40' is address of the controller and '10.131.29.42' is the address of the compute node, yes? Is the former publicly accessible from the same place you're browsing the web UX? I would assume not, given it's a private network IP. If not, this needs to be updated. This is the value returned from the API which is in turn used by Horizon iirc.
YES, the private internal IIP address scheme is what people hit horizon dashboard on. The clients are am on a private internal network.
Another thing to try is to connect the server directly using a local SPICE or VNC client. The IP address will be the IP of the compute host, while the port can be seen in the XML for each individual instance. This isn't a long-term solution but it's another way to rule out potentially faulty or misconfigured components.
I suspect the issue is with the address used for '[spice] html5proxy_base_url'. I would suggest using the 'openstack console url show' command to get a URL to access the console without needing to invoke Horizon. If you're able to access this URL then the issue is instead somewhere in Horizon. If not, it's definitely nova. You can also enable debug-level logging for the nova-spicehtml5proxy and nova-novncproxy services to ensure you're actually seeing a connection from your browser.
I will check openstack console url show
It says I need a <server>, but no matter what I put in, I get "no server name or ID of that exists".
Huh, perhaps I am loosing my mind? I can login to horizon and spin up VMs, but this is blank:
openstack server list
As I was trying different server names on openstack console url show.
This sounds like you are using a different tenant/project and possibly different user for the CLI vs. the web UI. If you're using a stackrc file, check the value of the 'OS_PROJECT_ID' and 'OS_PROJECT_NAME' environment variables against what's showing in your web UI (top right corner, iirc). If you're using a 'clouds.yaml' file, check the value of 'clouds.$cloud.auth.project_id' and 'clouds.$cloud.auth.project_name' against same.
Stephen
participants (4)
-
Matthias Runge
-
number9
-
Stephen Finucane
-
Thomas Goirand