Open Stack

Dear OpenStack community,


The short version: We are proposing a new set of use cases for OpenStack and a set of
changes to enable a multi-landlord cloud model, where multiple service
providers can cooperate (and compete) to stand up services in a single
cloud. We had great feedback from the community in the summit, and
came away with two strong messages: 1) this is a radical enough change
that we should do an end-to-end proof of concept, and 2) we should
post to this list what we are doing to make it visible to the broader
developer community; fully solving the problems of a multi-landlord
cloud will impact more projects than we understand today. We hope to
have available prototypes of the key enabling changes by the keystone
mid-cycle and an end-to-end demo by the Tokyo summit.  

Two additional points: 

 1. Solving the problems of landlords that don't trust each other
    also brings defense in depth for a single provider cloud;
    potentially preventing an exploit of one service from
    compromising an entire cloud.

 2. This work strongly relates to resource federation work that is
    ongoing in OpenStack, and is complementary to, and being persued
    in the context of the recently annouced Mercador project.

We, of course, welcome participating by other developers interested in
working with us on this through the Mercador project or by contacting
us as per info below.

The long version:
----------------------

All current clouds are stood up by a single company or organization,
creating a vertically integrated monopoly.  Any competition is between
entire clouds and is limited by the customer's ability to move their
data over the connectivity between clouds.  We think an alternative
model is possible, where different organizations can stand up
individual services in the same data centers, and the customer (or
intermediaries acting on their behalf) can pick and choose between
them.  We call this model of having multiple landlords in a cloud an
Open Cloud eXchange (OCX)
(http://www.cs.bu.edu/fac/best/res/papers/ic14.pdf <http://www.cs.bu.edu/fac/best/res/papers/ic14.pdf>).

The OCX model would enable more innovation by technology companies,
enable cloud research and provide more choice to cloud consumers. We
are developing this model in a new public cloud, the Massachusetts
Open Cloud (MOC), that is just being started in the MGHPCC data center
(http://www.mghpcc.org <http://www.mghpcc.org/>) shared between Boston University, Harvard
University, the Massachusetts Institute of Technology, Northeastern
University, and the University of Massachusetts.  Some use cases being
explored in the context of the MOC illustrate the potential of this
model:

o Harvard and MIT both stand up their own OpenStack cloud for
 students, but provide resources on-demand to the MOC that can be used
 by researchers that collaborate between the universities and by
 external users.  
o A storage company stands up a new innovative block storage service
 on a few racks in the MGHPCC, operates it themselves, and makes it
 available to users of the MOC and/or the individual university
 clouds.  The storage company is in total control of price,
 automation, and SLA for the service, and users can choose if they
 want to use the service.
o A company stands up a new compute service with innovative hardware
 (e.g., FPGAs, crypto accellerator) or pricing model.  A customer
 with a two Petabyte disk volume can switch to using that compute
 without having to move the data.
o A research group at Boston University and Northeastern develops a
 highly elastic compute service that can checkpoint 1000s of servers
 in seconds into SSD, and broadcast provision a distributed
 application to allow an interactive medical imaging application that
 wants 1000s of servers for a few seconds. 
o A security sensitive life sciences company exploits software from
 the MACS project (http://www.bu.edu/hic/research/macs/ <http://www.bu.edu/hic/research/macs/>) to
 distribute their data across two storage services from non-colluding
 providers.  The data is accessed from a Nova compute service running
 on a trusted compute platform developed collaboratively with
 Intel. Since all services are deployed in the same datacenter, the
 computation is efficient.
o Students in a cloud computing course offered by Northeastern and
 Boston University faculty
 (https://okrieg.github.io/EC500/index.html <https://okrieg.github.io/EC500/index.html>) develop and stand up an
 experimental proxy service for open stack services that provides
 users of the MOC a Swift service that combines the inventory of
 multiple underlying Swift services.

While no existing cloud stack can support the OCX model out of the
box, OpenStack is much closer than anything else, and we have been
exploring what changes will be required to enable this model
(http://open.bu.edu/handle/2144/11214 <http://open.bu.edu/handle/2144/11214>) and worked with our partners
in the community to submit a number of (now dated) blueprints to
handle our new use cases:
 o https://review.openstack.org/#/c/123782 <https://review.openstack.org/#/c/123782>
 o https://review.openstack.org/#/c/132623 <https://review.openstack.org/#/c/132623>
 o https://review.openstack.org/#/c/123726 <https://review.openstack.org/#/c/123726>

We believe solutions to the problems of the OCX model will improve
OpenStack generally from a security and reliability
perspective. Solving the problems of multiple providers/landlords that
don't trust each other also brings defense in depth for a single
provider cloud; potentially preventing an exploit of one service from
compromising an entire cloud.

We first described the idea of an OCX in the Atlanta summit
(https://www.youtube.com/watch?v=yuUb7XWnOcw <https://www.youtube.com/watch?v=yuUb7XWnOcw>) and the MOC project was
funded in December of 2014.  At the OpenStack summit in Vancouver we
had the amazing experience of the community embracing the idea, and
helping to educate us on much better solutions for the use cases.  For
example, based on the conversations we now have a scheme to build the
federated authorization on top of the recent federated authentication
capabilities of Keystone.  Also, we will be modifying Horizon rather
than developing a new UI. We will, of course, be updating the
blueprints and developing specs.

We wanted to send out this note for three reasons.  First, we wanted
to thank the community for all the feedback we got in the summit. We
were blown away by how open you are to new ideas & problems and your
patience in educating new members when we propose dumb solutions :).
Second, we had the strong advice from a number of developers that we
should send a note to this list letting a broader audience know about
the project; decisions are being made by many different groups that
will make the model more or less practical, and we understand that its
important to get the new use cases out there.  Third, we wanted to
announce that based on all the feedback we think we (who are a larger
team than before the summit :) now have a clear path to a
prototype. We hope to have available prototypes of many key enabling
changes by mid-cycle and an end-to-end demo by the Tokyo summit.  We
will be doing this work in the context of the newly annouced Mercador
project <https://wiki.openstack.org/wiki/Mercador <https://wiki.openstack.org/wiki/Mercador>>.  The Mercador
focus was originally on federation of resourcs between untrusting
single provider clouds.  When the two teams met at the summit, we
realized that the two projects solve orthogonal but complementary
problems, and we decided by joining forces we can help ensure that the
(still orthogonal) development efforts don't come up with solutions
that are incompatible.

Thanks again, it was an awesome experience meeting many of you in
Vancouver.
   -- the MOC/OCX gang ++

web:   http://massopencloud.org <http://massopencloud.org/>
email: moc-team-list at bu.edu <mailto:moc-team-list at bu.edu>
irc:   #moc on freenode
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150610/ba1868cc/attachment.html>