Open Stack

@malini-k-bhandaru
> I read the bug report and it mentions noVNC and SPICE, so I incorporated both in the messaging.

Good call.  I've adjusted some wording slightly, but the additions you
made look good overall.

> Which is the entity that becomes unresponsive?
> 1) The noVNC Proxy host? (the middle wheel in bottom that hosts the Nova console auth)
> Or
> 2) The compute node, top right?

The responsiveness of the nova-novncproxy service is what we are most
concerned about.  It is true that a host running nova-compute compute
node has to deal with the incoming VNC connections as well, but there
are usually many compute nodes and instances are balanced across them.
If the nova-novncproxy service is the bottleneck, then the compute node
itself won't become non-responsive.

> What is meant by "no amplification" ?

As I understand it, it's a when the volume of traffic in an attack is
amplified.  This typically involves getting many other services to
reflect traffic at the target.  For a real world example, lookup "smurf
attack".

> That means damage is limited right, then answer should be (2) above.

It's not limited (which is the main issue here).  It just means it's not
amplified.  That is, the attacker needs to be able to open enough
connections themself to overcome the resources of the system running
nova-novncproxy.

> But if it is two, each compute node would need to be listed and rate
limited?

As I mentioned above, the nova-novncproxy service is the main concern.
It wouldn't hurt to mention that a compute node could be affected too.
I've just added this to the note draft on the wiki.

-- 
You received this bug notification because you are a member of OpenStack
Security Group, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1227575

Title:
  DoS style attack on noVNC server can lead to service interruption or
  disruption

Status in OpenStack Compute (Nova):
  In Progress
Status in OpenStack Security Notes:
  New

Bug description:
  There is no limiting on the number of VNC sessions that can be created for a single user's VNC token.
  Any attempt to create multiple (say hundreds or thousands) of websocket connections to the VNC server
  results in many connection timeouts. Due to these connection timeout error, other users trying to access their 
  VM's VNC console cannot do so.

  A sample script that tries to create 100,000 connections to Nova noVNC proxy, shows timeout errors
  Script: http://paste.openstack.org/show/47254/

  Script output.... connections get timed out after a while
  -------------------
  ....
  ..

  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  --------------------

  Impact:
  1. Many of the sessions timeout. Any attempt to open other sessions also intermittently fail.
  This can cause serious problems to users already having a running VNC session or trying to create new sessions.

  2. The overall performance and response times of other nova services running on the novnc host, using tcp protocol
  also gets affected after the connection timeout errors.

  For example:
  Before running the sumulate thousands of connections program:
  $ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5  novnc
  +-------+---------------------------------------------------------------------------------+
  | Type  | Url                                                                             |
  +-------+---------------------------------------------------------------------------------+
  | novnc | http://10.2.3.102:6080/vnc_auto.html?token=e776dd33-422f-4b56-9f98-e317410d0212 |
  +-------+---------------------------------------------------------------------------------+

  real    0m0.751s
  user    0m0.376s
  sys     0m0.084s

  rohit at precise-dev-102:~/tools/websocket-client-0.7.0$

  After running the program, the response time is quite high:
  $ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5  novnc

  +-------+---------------------------------------------------------------------------------+
  | Type  | Url                                                                             |
  +-------+---------------------------------------------------------------------------------+
  | novnc | http://10.2.3.102:6080/vnc_auto.html?token=6865d675-d852-478b-b1ee-457b092f11b9 |
  +-------+---------------------------------------------------------------------------------+

  real    3m9.231s
  user    0m0.424s
  sys     0m0.108s

  
  Possible solutions:
  1. Allow just 1 session per instance, and raise a new exception, say, VNCSessionAlreadyExists to reject multiple
  connections for the same token, and return an error code to the user.
  2. Make the number of sessions allowed per instance configurable, limited by some count of sessions.

  However, both of these solutions may need to override and modify the do_proxy() method of websockify's WebSocketProxy class, 
  which can lead to maintenance issues.

  Another possible solution would be to implement some kind of callback function in websockify, to which we can pass the token
  for reconnection. This would first need contribution to the websockify project code, and then update Nova.

To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1227575/+subscriptions