Open Stack

I'll make those corrections, thanks Rob.

-----Original Message-----
From: "Clark, Robert Graham" <robert.clark at hp.com>
Sent: ‎3/‎8/‎2014 12:51 PM
To: "Sriram Subramanian" <sriram at sriramhere.com>; "Bug 1227575" <1227575 at bugs.launchpad.net>; "openstack-security at lists.openstack.org" <openstack-security at lists.openstack.org>
Subject: RE: [Openstack-security] [Bug 1227575] Re: DoS style attack onnoVNCserver can lead to service interruption or disruption

The OSSN looks great, but I think perhaps the summary could be tweaked to be a little shorter and maybe flow a little better…
 
There is currently no limit to the number of noVNC or SPICE console sessions that can be established by a single user. The console host has limited resources and an attacker launching many sessions may be able to exhaust the available resources, resulting in a Denial of Service (DoS) condition.  
 
Other than that it’s hot-to-trot, as they say.
 
-Rob
 
From: Sriram Subramanian [mailto:sriram at sriramhere.com] 
Sent: 08 March 2014 06:03
To: Bug 1227575; openstack-security at lists.openstack.org
Subject: Re: [Openstack-security] [Bug 1227575] Re: DoS style attack on noVNCserver can lead to service interruption or disruption
 
Thanks Nathan, good to publish!



From: Nathan Kinder
Sent: ‎3/‎7/‎2014 8:33 PM
To: openstack-security at lists.openstack.org
Subject: [Openstack-security] [Bug 1227575] Re: DoS style attack on noVNCserver can lead to service interruption or disruption
Here's an updated OSSN draft that covers all of the issues mentioned in
my previous comment (in the appropriate e-mail format).  I also removed
the reference to rate-limiting in the Security Guide, as it doesn't
actually contain any information on how to do rate limiting.  It only
mentioned that QOSaaS plans to offer rate limiting, which isn't
applicable here.

If this looks fine to everyone else, I'll go ahead and publish it.

-----------------------------------------------------------------------------------------
DoS style attack on noVNC server can lead to service interruption or disruption
---

### Summary ###
There is currently no limit on the number of noVNC or SPICE console
sessions that can be established against a single user token. This
enables launching a Denial of Service (DoS) style attack by establishing
many console sessions against a single virtual machine instance through
the console proxy. This can cause instance access timeouts and general
service response degradation on the console host.

### Affected Services / Software ###
Horizon, Nova, noVNC proxy, SPICE console, Grizzly, Havana

### Discussion ###
Currently with a single user token, no restrictions are enforced on the
number or frequency of noVNC or SPICE console sessions that may be
established.  While a user can only access their own virtual machine
instances, resources can be exhausted on the console proxy host by
creating an excessive number of simultaneous console sessions.  This can
result in timeouts for subsequent connection requests to instances using
the same console proxy.  Not only would this prevent the user from
accessing their own instances, but other legitimate users would also be
deprived of console access.  Further, other services running on the
noVNC proxy and Compute hosts may degrade in responsiveness.

By taking advantage of this lack of restrictions around noVNC or SPICE
console connections, a single user could cause the console proxy
endpoint to become unresponsive, resulting in a Denial Of Service (DoS)
style attack.  It should be noted that there is no amplification effect.

### Recommended Actions ###
For current stable OpenStack releases (Grizzly, Havana), users need to
workaround this vulnerability by using rate-limiting proxies to cover
access to the noVNC proxy service.  Rate-limiting is a common mechanism
to prevent DoS and Brute-Force attacks.

For example, if you are using a proxy such as Repose, enable the rate
limiting feature by following these steps:

  https://repose.atlassian.net/wiki/display/REPOSE/Rate+Limiting+Filter

Future OpenStack releases are looking to add the ability to restrict
noVNC and SPICE console connections.

### Contacts / References ###
This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0008
Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/1227575
OpenStack Security ML : openstack-security at lists.openstack.org
OpenStack Security Group : https://launchpad.net/~openstack-ossg

-- 
You received this bug notification because you are a member of OpenStack
Security Group, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1227575

Title:
  DoS style attack on noVNC server can lead to service interruption or
  disruption

Status in OpenStack Compute (Nova):
  In Progress
Status in OpenStack Security Notes:
  New

Bug description:
  There is no limiting on the number of VNC sessions that can be created for a single user's VNC token.
  Any attempt to create multiple (say hundreds or thousands) of websocket connections to the VNC server
  results in many connection timeouts. Due to these connection timeout error, other users trying to access their 
  VM's VNC console cannot do so.

  A sample script that tries to create 100,000 connections to Nova noVNC proxy, shows timeout errors
  Script: http://paste.openstack.org/show/47254/

  Script output.... connections get timed out after a while
  -------------------
  ....
  ..

  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  Received 'RFB 003.008
  '
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  Creating Connection
  Receiving...
  timed out
  --------------------

  Impact:
  1. Many of the sessions timeout. Any attempt to open other sessions also intermittently fail.
  This can cause serious problems to users already having a running VNC session or trying to create new sessions.

  2. The overall performance and response times of other nova services running on the novnc host, using tcp protocol
  also gets affected after the connection timeout errors.

  For example:
  Before running the sumulate thousands of connections program:
  $ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5  novnc
  +-------+---------------------------------------------------------------------------------+
  | Type  | Url                                                                             |
  +-------+---------------------------------------------------------------------------------+
  | novnc | http://10.2.3.102:6080/vnc_auto.html?token=e776dd33-422f-4b56-9f98-e317410d0212 |
  +-------+---------------------------------------------------------------------------------+

  real    0m0.751s
  user    0m0.376s
  sys     0m0.084s

  rohit at precise-dev-102:~/tools/websocket-client-0.7.0$

  After running the program, the response time is quite high:
  $ time nova get-vnc-console c1b093a3-f53b-4282-b89c-e68f0fa1b6e5  novnc

  +-------+---------------------------------------------------------------------------------+
  | Type  | Url                                                                             |
  +-------+---------------------------------------------------------------------------------+
  | novnc | http://10.2.3.102:6080/vnc_auto.html?token=6865d675-d852-478b-b1ee-457b092f11b9 |
  +-------+---------------------------------------------------------------------------------+

  real

[The entire original message is not included.]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-security/attachments/20140308/9d646899/attachment.html>