Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Live migration instructions recommend unsecured libvirt remote access
- ---

### Summary ###
When using the KVM hypervisor with libvirt on OpenStack Compute nodes,
live migration of instances from one Compute server to another requires
that the libvirt daemon is configured for remote network connectivity.
The libvirt daemon configuration recommended in the OpenStack
Configuration Reference manual configures libvirtd to listen for
incoming TCP connections on all network interfaces without requiring any
authentication or using any encryption.  This insecure configuration
allows for anyone with network access to the libvirt daemon TCP port on
OpenStack Compute nodes to control the hypervisor through the libvirt
API.

### Affected Services / Software ###
Nova, Compute, KVM, libvirt, Grizzly, Havana, Icehouse

### Discussion ###
The default configuration of the libvirt daemon is to not allow remote
access.  Live migration of running instances between OpenStack Compute
nodes requires libvirt daemon remote access between OpenStack Compute
nodes.

The libvirt daemon should not be configured to allow unauthenticated
remote access.  The libvirt daemon  has a choice of 4 secure options for
remote access over TCP.  These options are:

 - SSH tunnel to libvirtd's UNIX socket
 - libvirtd TCP socket, with GSSAPI/Kerberos for auth+data encryption
 - libvirtd TCP socket, with TLS for encryption and x.509 client
   certificates for authentication
 - libvirtd TCP socket, with TLS for encryption and Kerberos for
   authentication

It is not necessary for the libvirt daemon to listen for remote TCP
connections on all interfaces.  Remote network connectivity to the
libvirt daemon should be restricted as much as possible.  Remote
access is only needed between the OpenStack Compute nodes, so the
libvirt daemon only needs to listen for remote TCP connections on the
interface that is used for this communication.  A firewall can be
configured to lock down access to the TCP port that the libvirt daemon
listens on, but this does not sufficiently protect access to the libvirt
API.  Other processes on a remote OpenStack Compute node might have
network access, but should not be authorized to remotely control the
hypervisor on another OpenStack Compute node.

### Recommended Actions ###
If you are using the KVM hypervisor with libvirt on OpenStack Compute
nodes, you should review your libvirt daemon configuration to ensure
that it is not allowing unauthenticated remote access.

Remote access to the libvirt daemon via TCP is configured by the
"listen_tls", "listen_tcp", and "auth_tcp" configuration directives.  By
default, these directives are all commented out.  This results in remote
access via TCP being disabled.

If you do not need remote libvirt daemon access, you should ensure that
the following configuration directives are set as follows in the
/etc/libvirt/libvirtd.conf configuration file.  Commenting out these
directives will have the same effect, as these values match the internal
defaults:

- ---- begin example libvirtd.conf snippet ----
listen_tls = 1
listen_tcp = 0
auth_tcp = "sasl"
- ---- end example libvirtd.conf snippet ----

If you need to allow remote access to the libvirt daemon between
OpenStack Compute nodes for live migration, you should ensure that
authentication is required.  Additionally, you should consider enabling
TLS to allow remote connections to be encrypted.

The following libvirt daemon configuration directives will allow for
unencrypted remote connections that use SASL for authentication:

- ---- begin example libvirtd.conf snippet ----
listen_tls = 0
listen_tcp = 1
auth_tcp = "sasl"
- ---- end example libvirtd.conf snippet ----

If you want to require TLS encrypted remote connections, you will have
to obtain X.509 certificates and configure the libvirt daemon to use
them to use TLS.  Details on this configuration are in the libvirt
daemon documentation.  Once the certificates are configured, you should
set the following libvirt daemon configuration directives:

- ---- begin example libvirtd.conf snippet ----
listen_tls = 1
listen_tcp = 0
auth_tls = "none"
- ---- end example libvirtd.conf snippet ----

When using TLS, setting the "auth_tls" configuration directive to "none"
uses X.509 client certificates for authentication.  You can additionally
require SASL authentication by setting the following libvirt daemon
configuration directives:

- ---- begin example libvirtd.conf snippet ----
listen_tls = 1
listen_tcp = 0
auth_tls = "sasl"
- ---- end example libvirtd.conf snippet ----

When using TLS, it is also necessary to configure the OpenStack Compute
nodes to use a non-default URI for live migration.  This is done by
setting the following configuration directive in /etc/nova/nova.conf:

- ---- begin example nova.conf snippet ----
live_migration_uri=qemu+tls://%s/system
- ---- end example nova.conf snippet ----

For more details on libvirt daemon remote URI formats, please see the
following libvirt daemon documentation:

  http://libvirt.org/remote.html#Remote_URI_reference

For details on configuring SASL authentication and X.509 certificates
for the libvirt daemon, please consult the libvirt daemon documentation
at the following locations:

  http://libvirt.org/remote.html
  http://libvirt.org/auth.html

When configuring the libvirt daemon for authentication, it is also
important to configure authorization to restrict remote access to your
OpenStack Compute nodes.  For example, if you don't configure
authorization, any X.509 client certificate that is signed by your
issuing CA will be allowed access.  When using SASL/GSSAPI for Kerberos
authentication, any client with a valid TGT will be granted access.
Lack of authorization can allow unintended remote access.  The libvirt
daemon documentation should be consulted for details on configuring
authorization.

In addition to requiring authentication for remote access to the libvirt
daemon on your OpenStack Compute nodes, it is also recommended to
restrict network access such that connectivity is only allowed between
the Compute nodes.

The first thing that should be done is to restrict the network
interfaces that the libvirt daemon listens on for remote connections.
By default, the libvirt daemon listens on all interfaces when remote
access is enabled.  This can be restricted by setting the following
configuration directive in /etc/libvirt/libvirtd.conf:

- ---- begin example libvirtd.conf snippet ----
listen_addr = <IP address or hostname>
- ---- end example libvirtd.conf snippet ----

Migration in the libvirt daemon also uses a range of ephemeral ports by
default.  The connections to these ephemeral ports are not authenticated
or encrypted.  It is possible to tunnel the migration traffic over the
regular libvirt daemon remote access port, which will use the
authentication and encryption settings that you have defined for that
port.  It is recommended that you do this for the additional security
that it provides.  To enable tunnelling of the migration traffic, you
must tell your OpenStack Compute nodes to set the VIR_MIGRATE_TUNNELLED
flag for live migration.  This is done by setting the following
directive in /etc/nova/nova.conf:

- ---- begin example nova.conf snippet ----
live_migration_flag=VIR_MIGRATE_UNDEFINE_SOURCE,
VIR_MIGRATE_PEER2PEER, VIR_MIGRATE_TUNNELLED
- ---- end example nova.conf snippet ----

The tunnelling of migration traffic described above does not apply to
live block migration.  Live block migration is currently only possible
over ephemeral ports.

If you choose to use the ephemeral migration ports, there are a few
things that you should be aware of.  Unfortunately, there is no way to
restrict the network interfaces that these ephemeral ports will listen
on in libvirt versions prior to 1.1.4.  If you are running version 1.1.4
or later of the libvirt daemon, you can set the following directive in
/etc/libvirt/qemu.conf to specify what interfaces are used for the
ephemeral migration ports:

- ---- begin example qemu.conf snippet ----
migration_address = <IP address>
- ---- end example qemu.conf snippet ----

It is also recommended to configure the firewall on each OpenStack
Compute node to only allow other Compute nodes to access the ports that
are used for remote access to the libvirt daemon.  By default, this is
port 16514 for TLS and 16509 for unencrypted TCP.

Additionally, migration over ephemeral ports uses a port range of
49152-49215.  You will need to allow your OpenStack Compute nodes to
communicate with each other over these ports if you choose not to tunnel
the migration traffic over the libvirt daemon remote access port.

You can check what ports you have configured for the libvirt daemon by
looking at the following configuration directives:

  tls_port (libvirtd.conf)
  tcp_port (libvirtd.conf)
  migration_port_min (qemu.conf)t
  migration_port_max (qemu.conf)

If you are running a version of the libvirt daemon older than 1.1.4 and
you want to perform live block migration, you will need to allow your
OpenStack Compute nodes to communicate over port range 5900-49151.  If
you are running version 1.1.4 or later of the libvirt daemon, the
regular ephemeral migration port range is used for live block migration.

Please consult the documentation for your firewall software for
instructions on configuring the appropriate firewall rules.

### Contacts / References ###
This OSSN: https://wiki.openstack.org/wiki/OSSN/OSSN-0007
Original LaunchPad Bug :
https://bugs.launchpad.net/openstack-manuals/+bug/1287194
OpenStack Security ML : openstack-security at lists.openstack.org
OpenStack Security Group : https://launchpad.net/~openstack-ossg
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.13 (GNU/Linux)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTGM4qAAoJEJa+6E7Ri+EVWYkH/2vIVso0+rFiHOZGWBETD22D
3PMT9ozf/DKtfA6QwH4vLYuBZxTPealng4eO41xtEoyw/I8k9Gbwu60BrNQSPPrH
s4s3Cohn98CauI+fL9ZzF5+s5wCAIsxzJaSCnwMeyk/cxjb4IbRhP+bYCC+Dx9GF
Q5eJ0CMP4hedX8U7St23lDnnGnVmWZd93GsiCWZjHXyKkpveudDci6b8Q9g1tYrb
7CLtTywUSR00+m1x0F4MV5PiyTVjn+wIv6IqJklv83kNHxurtibLBfAfjHOfli84
8PTNGYR8vgZ5FSY+2SsEUv5F6OECk3wEH0QVEXGji9Y9bHiLAEl4dPMfL4i2PhA=
=ZaH2
-----END PGP SIGNATURE-----