[Openstack-security] [Bug 1227575] Re: DoS style attack on noVNC server can lead to service interruption or disruption
Meghal Gosalia
meghal at yahoo-inc.com
Thu Mar 13 18:16:46 UTC 2014
I have spice enabled in my havana environment and am using spice-html5proxy to connect to the spice server.
If I use an URL with same token multiple times, previous connection drops and only last session stays active.
Old tab shows "Error: Unexpected close while ready" in console.log under spice console on the browser.
And latest tab has active connection to the spice server.
Here is what I see in qemu logs, when I use same token again to connect
to the server -
red_client_destroy: destroy client with #channels 4
red_dispatcher_disconnect_display_peer:
red_channel_client_disconnect: 0x7f42302a81a0 (channel 0x7f423021d0b0 type 2 id 0)
red_channel_client_disconnect: 0x7f42302a81a0 (channel 0x7f423021d0b0 type 2 id 0)
red_channel_client_disconnect: 0x7f42714a1dc0 (channel 0x7f4270efc460 type 3 id 0)
red_channel_client_disconnect: 0x7f42714a1dc0 (channel 0x7f4270efc460 type 3 id 0)
red_dispatcher_disconnect_cursor_peer:
red_channel_client_disconnect: 0x7f42302660c0 (channel 0x7f423021d670 type 4 id 0)
red_channel_client_disconnect: 0x7f42302660c0 (channel 0x7f423021d670 type 4 id 0)
red_channel_client_disconnect: 0x7f42714740f0 (channel 0x7f4270ef2770 type 1 id 0)
main_channel_client_on_disconnect: rcc=0x7f42714740f0
red_channel_client_disconnect: 0x7f42714740f0 (channel 0x7f4270ef2770 type 1 id 0)
main_channel_link: add main channel client
main_channel_handle_parsed: net test: latency 19.735000 ms, bitrate 25127601 bps (23.963548 Mbps)
red_dispatcher_set_cursor_peer:
inputs_connect: inputs channel client create
So, does this mean that I am using certain qemu configuration (I do not see any) which does not allow to share connections?
If qemu is not allowing multiple clients connecting to spice server, will I still see the DoS attack?
--
You received this bug notification because you are a member of OpenStack
Security Group, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1227575
Title:
DoS style attack on noVNC server can lead to service interruption or
disruption
Status in OpenStack Compute (Nova):
In Progress
Status in OpenStack Security Notes:
Fix Released
Bug description:
There is no limiting on the number of VNC sessions that can be created for a single user's VNC token.
Any attempt to create multiple (say hundreds or thousands) of websocket connections to the VNC server
results in many connection timeouts. Due to these connection timeout error, other users trying to access their
VM's VNC console cannot do so.
A sample script that tries to create 100,000 connections to Nova noVNC proxy, shows timeout errors
Script: http://paste.openstack.org/show/47254/
Script output.... connections get timed out after a while
-------------------
....
..
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
Received 'RFB 003.008
'
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
Creating Connection
Receiving...
timed out
--------------------
Impact:
1. Many of the sessions timeout. Any attempt to open other sessions also intermittently fail.
This can cause serious problems to users already having a running VNC session or trying to create new sessions.
2. The overall performance and response times of other nova services running on the novnc host, using tcp protocol
also gets affected after the connection timeout errors.
For example:
Before running the sumulate thousands of connections program:
$ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5 novnc
+-------+---------------------------------------------------------------------------------+
| Type | Url |
+-------+---------------------------------------------------------------------------------+
| novnc | http://10.2.3.102:6080/vnc_auto.html?token=e776dd33-422f-4b56-9f98-e317410d0212 |
+-------+---------------------------------------------------------------------------------+
real 0m0.751s
user 0m0.376s
sys 0m0.084s
rohit at precise-dev-102:~/tools/websocket-client-0.7.0$
After running the program, the response time is quite high:
$ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5 novnc
+-------+---------------------------------------------------------------------------------+
| Type | Url |
+-------+---------------------------------------------------------------------------------+
| novnc | http://10.2.3.102:6080/vnc_auto.html?token=6865d675-d852-478b-b1ee-457b092f11b9 |
+-------+---------------------------------------------------------------------------------+
real 3m9.231s
user 0m0.424s
sys 0m0.108s
Possible solutions:
1. Allow just 1 session per instance, and raise a new exception, say, VNCSessionAlreadyExists to reject multiple
connections for the same token, and return an error code to the user.
2. Make the number of sessions allowed per instance configurable, limited by some count of sessions.
However, both of these solutions may need to override and modify the do_proxy() method of websockify's WebSocketProxy class,
which can lead to maintenance issues.
Another possible solution would be to implement some kind of callback function in websockify, to which we can pass the token
for reconnection. This would first need contribution to the websockify project code, and then update Nova.
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1227575/+subscriptions
More information about the Openstack-security
mailing list