[openstack-dev] [magnum][app-catalog][all] Build unified abstraction for all COEs
Fox, Kevin M
Kevin.Fox at pnnl.gov
Wed Apr 20 16:08:21 UTC 2016
So, location of unified api is an interesting topic... Can things be arranged in such a way that the abstraction largely exists only on the client in its own project?
So, what about a change of ideas....
The three pain points in the abstraction are:
* Authentication
* Connectivity
* Differences in COE features/api that need abstracting.
Authentication issues: All OpenStack services use the same creds. Its painful if you ever have to use another set. How does a client ask Magnum for creds for a COE if it doesn't have some. Its painful if doesn't use keystone tokens. Three ways to solve this... 1, add keystone auth support to the COEs and have Magnum always configure it. 2, Proxy the COE api through Magnum's api, and deal with authentication on the Magnum side using Keystone. 3, Have some kind of way to provision users/creds in each of the COE's and have a way to fetch the creds easily via Magnum's api.
Connectivity issues: The cloud ops ensure Magnum/Keystone/etc's apis are available. The COE's api is potentially in a tenant or private network, where it can be difficult to know how to get to properly. If you have two COE's in two different tenants both using 10.0.0.0/24 you may run into issues keeping them apart. So I can see two potential ways forward here. Again, proxying the native COE api requests through Magnum could solve the issue. The other would be mandating every COE have a floating ip, though I think there still are issues there with clouds with multiple external networks and connectivity (I have a cloud with multiple external networks, so I know they are real :)
So, what do folks think about a proxy? I think all three COE API's are rest based? If so, they could be grafted into magum api, something like /v1/coeproxy/<id_of_coe>/.
A unified client could then use keystone auth and the magnum api to talk to the COE, and then all the rest of the Abstraction nasties are all Features/API differences and totally on the client to figure out. This can be a non Magnum project if its too distasteful.
Thanks,
Kevin
________________________________________
From: Keith Bray [keith.bray at RACKSPACE.COM]
Sent: Tuesday, April 19, 2016 10:12 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [magnum][app-catalog][all] Build unified abstraction for all COEs
Sure… I can clarify with a few additional thoughts:
.1) I wouldn’t recommend that it be required for the operator to offer
this API. Representing a view of both providing managed services for
private cloud customer-on-premise installations of upstream OpenStack and
as a business owner with responsibility to operate Magnum for internal
usage within my own employer, I would prefer not to have to operate and
service a unified abstraction API that obfuscates all the benefit of
choice of the native COEs, which is the choice being provided to the end
user who is specifically selecting one COE over another when they
instantiate a bay (unless they pick the “Default” operator choice). Maybe
a unified abstraction API is a separate project? OpenStack services get
complicated very quickly and try to do too much. At a minimum, I would
recommend it be an optional API, not required, and any overhead of
database or other necessary service components should be minimized to not
impact operators who do not want to offer it because it negates the point
of COE choice. My ideal state is it would be a separate project entirely.
.2) I’d like for folks who want the lowest common denominator API to chime
in with why they want it, and whether they need it to be part of Magnum or
not. I don’t intend to argue with folks who want it… I assume their
reasons are justified, but I would want to find out why it needs to be
part of the Magnum API. Offering choice in COEs and then getting out of
the way (which I believe Magnum should do) is at odds with abstracting the
differentiation of the CEO choice via a unified API. If there aren’t good
arguments for the "why a unified API needs to be integrated in Magnum",
then have it be separate from a code perspective and not required for
running the Magnum service. When we talk about APIs and whether a service
is supported by one vendor or another, it is generally easiest to think
about the entire API; The API is either supported in its entirety or the
service isn’t compatible with OpenStack. If some folks believe a lowest
common denominator API should exist, but there aren’t compelling arguments
for why it must be a required part of the Magnum API then we should
probably consider them as separate projects. At this point, I am not
compelled to be in favor of integrating a unified API in Magnum when doing
so is a fundamentally different direction than the route Magnum has been
headed down which. By offering choice of COE, and trying not to get in
the way of that, Magnum provides relevant choice of platform to a very
rapidly changing technology landscape.
Thank you for asking for clarification. I’d really like to hear thoughts
from anyone who wants the unified API as to why it would need to be part
of Magnum, especially when doing so means chasing rapidly changing
technologies (hard to keep up with continued abstraction) and not offering
the deep value of their differentiation.
Kind regards,
-Keith
On 4/19/16, 8:58 PM, "Hongbin Lu" <hongbin.lu at huawei.com> wrote:
>I am going to clarify one thing. Users will always have access to native
>APIs provided by individual COEs, regardless of the existence of the
>common abstraction. In other words, the proposed common abstraction layer
>is an addition, not a replacement.
>
>Best regards,
>Hongbin
>
>> -----Original Message-----
>> From: Keith Bray [mailto:keith.bray at RACKSPACE.COM]
>> Sent: April-19-16 7:17 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [magnum][app-catalog][all] Build unified
>> abstraction for all COEs
>>
>> I would recommend against implementing a lowest common denominator API
>> for
>> the COEs ³right now.² It¹s too early to tell if the COEs are going to
>> be
>> seen as a commodity (where in the long run they may all perform
>> relatively equal for the majority of workloads < in which case why do
>> you care to have choice in COE?), or continue to be
>> specialized/differentiated. If you assume having choice to provision
>> more than one COE using the same system is valuable, then it is logical
>> that users value the differentiation in the COEs in some way. If they
>> are differentiated, and you value that, then you likely want to avoid
>> the lowest-common-demonitator API because that abstracts you from the
>> differentiation that you value.
>>
>> Kind regards,
>> -Keith
>>
>>
>>
>> On 4/19/16, 10:18 AM, "Hongbin Lu" <hongbin.lu at huawei.com> wrote:
>>
>> >Sorry, it is too late to adjust the schedule now, but I don't mind to
>> >have a pre-discussion here. If you have opinions/ideas on this topic
>> >but cannot attend the session [1], we'd like to have you inputs in
>> this
>> >ML or in the etherpad [2]. This will help to set the stage for the
>> session.
>> >
>> >For background, Magnum supports provisioning Container Orchestration
>> >Engines (COEs), including Kubernetes, Docker Swarm and Apache Mesos,
>> on
>> >top of Nova instances. After the provisioning, users need to use the
>> >native COE APIs to manage containers (and/or other COE resources). In
>> >the Austin summit, we will have a session to discuss if it makes sense
>> >to build a common abstraction layer for the supported COEs. If you
>> >think it is a good idea, it would be great to elaborate the details.
>> >For example, answering the following questions could be useful:
>> >* Which abstraction(s) you are looking for (i.e. container, pod)?
>> >* What are your use cases for the abstraction(s)?
>> >* How the native APIs provided by individual COEs doesn't satisfy your
>> >requirements?
>> >
>> >If you think it is a bad idea, I would love to hear your inputs as
>> well:
>> >* Why it is bad?
>> >* If there is no common abstraction, how to address the pain of
>> >leveraging native COE APIs as reported below?
>> >
>> >[1]
>> >https://www.openstack.org/summit/austin-2016/summit-
>> schedule/events/910
>> >2 [2]
>> >https://etherpad.openstack.org/p/newton-magnum-unified-abstraction
>> >
>> >Best regards,
>> >Hongbin
>> >
>> >> -----Original Message-----
>> >> From: Fox, Kevin M [mailto:Kevin.Fox at pnnl.gov]
>> >> Sent: April-18-16 6:13 PM
>> >> To: OpenStack Development Mailing List (not for usage questions);
>> >> Flavio Percoco
>> >> Cc: foundation at lists.openstack.org
>> >> Subject: Re: [openstack-dev] [OpenStack Foundation] [board][tc][all]
>> >> One Platform - Containers/Bare Metal? (Re: Board of Directors
>> >> Meeting)
>> >>
>> >> I'd love to attend, but this is right on top of the app catalog
>> meeting.
>> >> I think the app catalog might be one of the primary users of a cross
>> >> COE api.
>> >>
>> >> At minimum we'd like to be able to be able to store url's for
>> >> Kubernetes/Swarm/Mesos templates and have an api to kick off a
>> >> workflow in Horizon to have Magnum start up a new instance of of the
>> >> template the user selected.
>> >>
>> >> Thanks,
>> >> Kevin
>> >> ________________________________________
>> >> From: Hongbin Lu [hongbin.lu at huawei.com]
>> >> Sent: Monday, April 18, 2016 2:09 PM
>> >> To: Flavio Percoco; OpenStack Development Mailing List (not for
>> usage
>> >> questions)
>> >> Cc: foundation at lists.openstack.org
>> >> Subject: Re: [openstack-dev] [OpenStack Foundation] [board][tc][all]
>> >> One Platform - Containers/Bare Metal? (Re: Board of Directors
>> >> Meeting)
>> >>
>> >> Hi all,
>> >>
>> >> Magnum will have a fishbowl session to discuss if it makes sense to
>> >> build a common abstraction layer for all COEs (kubernetes, docker
>> >> swarm and mesos):
>> >>
>> >> https://www.openstack.org/summit/austin-2016/summit-
>> >> schedule/events/9102
>> >>
>> >> Frankly, this is a controversial topic since I heard agreements and
>> >> disagreements from different people. It would be great if all of you
>> >> can join the session and share your opinions and use cases. I wish
>> we
>> >> will have a productive discussion.
>> >>
>> >> Best regards,
>> >> Hongbin
>> >>
>> >> > -----Original Message-----
>> >> > From: Flavio Percoco [mailto:flavio at redhat.com]
>> >> > Sent: April-12-16 8:40 AM
>> >> > To: OpenStack Development Mailing List (not for usage questions)
>> >> > Cc: foundation at lists.openstack.org
>> >> > Subject: Re: [openstack-dev] [OpenStack Foundation]
>> >> > [board][tc][all] One Platform - Containers/Bare Metal? (Re: Board
>> >> > of Directors Meeting)
>> >> >
>> >> > On 11/04/16 16:53 +0000, Adrian Otto wrote:
>> >> > >Amrith,
>> >> > >
>> >> > >I respect your point of view, and agree that the idea of a common
>> >> > >compute API is attractive... until you think a bit deeper about
>> >> > >what
>> >> > that
>> >> > >would mean. We seriously considered a "global" compute API at the
>> >> > >time we were first contemplating Magnum. However, what we came to
>> >> > >learn through the journey of understanding the details of how
>> such
>> >> > >a thing would be implemented, that such an API would either be (1)
>> >> > >the lowest common denominator (LCD) of all compute types, or (2)
>> >> > >an exceedingly
>> >> > complex interface.
>> >> > >
>> >> > >You expressed a sentiment below that trying to offer choices for
>> >> > >VM, Bare Metal (BM), and Containers for Trove instances "adds
>> >> > >considerable
>> >> > complexity".
>> >> > >Roughly the same complexity would accompany the use of a
>> >> > >comprehensive compute API. I suppose you were imagining an LCD
>> >> > >approach. If that's what you want, just use the existing Nova API,
>> >> > >and load different compute drivers on different host aggregates.
>> A
>> >> > >single Nova client can produce VM, BM (Ironic), and Container
>> >> > >(lbvirt-lxc) instances all with a common API (Nova) if it's
>> >> configured in this way. That's what we do.
>> >> > >Flavors determine which compute type you get.
>> >> > >
>> >> > >If what you meant is that you could tap into the power of all the
>> >> > >unique characteristics of each of the various compute types
>> >> > >(through some modular extensibility framework) you'll likely end
>> >> > >up with complexity in Trove that is comparable to integrating
>> with
>> >> > >the
>> >> native
>> >> > >upstream APIs, along with the disadvantage of waiting for
>> >> > >OpenStack to continually catch up to the pace of change of the
>> >> > >various
>> >> upstream
>> >> > >systems on which it depends. This is a recipe for disappointment.
>> >> > >
>> >> > >We concluded that wrapping native APIs is a mistake, particularly
>> >> > >when they are sufficiently different than what the Nova API
>> >> > >already
>> >> offers.
>> >> > >Containers APIs have limited similarities, so when you try to
>> make
>> >> > >a universal interface to all of them, you end up with a really
>> >> > >complicated mess. It would be even worse if we tried to
>> >> > >accommodate
>> >> > all
>> >> > >the unique aspects of BM and VM as well. Magnum's approach is to
>> >> > >offer the upstream native API's for the different container
>> >> > >orchestration engines (COE), and compose Bays for them to run on
>> >> that
>> >> > >are built from the compute types that OpenStack supports. We do
>> >> > >this by using different Heat orchestration templates (and
>> >> > >conditional
>> >> > >templates) to arrange a COE on the compute type of your choice.
>> >> > >With that said,
>> >> > there
>> >> > >are still gaps where not all storage or network drivers work with
>> >> > >Ironic, and there are non-trivial security hurdles to clear to
>> >> safely
>> >> > use Bays composed of libvirt-lxc instances in a multi-tenant
>> >> > environment.
>> >> > >
>> >> > >My suggestion to get what you want for Trove is to see if the
>> >> > >cloud
>> >> > has
>> >> > >Magnum, and if it does, create a bay with the flavor type
>> >> > >specified
>> >> > for
>> >> > >whatever compute type you want, and then use the native API for
>> >> > >the
>> >> > COE
>> >> > >you selected for that bay. Start your instance on the COE, just
>> >> > >like you use Nova today. This way, you have low complexity in
>> >> > >Trove, and
>> >> > you
>> >> > >can scale both the number of instances of your data nodes
>> >> > >(containers), and the infrastructure on which they run (Nova
>> >> instances).
>> >> >
>> >> >
>> >> > I've been researching on this area and I've reached pretty much
>> the
>> >> > same conclusion. I've had moments of wondering whether creating
>> >> > bays is something Trove should do but I now think it should.
>> >> >
>> >> > The need of handling the native API is the part I find a bit
>> >> > painful as that means more code needs to happen in Trove for us to
>> >> > provide this provisioning facilities. I wonder if a common
>> >> > *library* would help here, at least to handle those "simple" cases.
>> >> > Anyway, I look forward to chatting with you all about this.
>> >> >
>> >> > It'd be great if you (and other magnum folks) could join this
>> session:
>> >> >
>> >> > https://etherpad.openstack.org/p/trove-newton-summit-container
>> >> >
>> >> > Thanks for chiming in, Adrian.
>> >> > Flavio
>> >> >
>> >> > >Regards,
>> >> > >
>> >> > >Adrian
>> >> > >
>> >> > >
>> >> > >
>> >> > >
>> >> > > On Apr 11, 2016, at 8:47 AM, Amrith Kumar <amrith at tesora.com>
>> >> > wrote:
>> >> > >
>> >> > > Monty, Dims,
>> >> > >
>> >> > > I read the notes and was similarly intrigued about the idea.
>> >> > > In
>> >> > particular,
>> >> > > from the perspective of projects like Trove, having a common
>> >> > Compute API is
>> >> > > very valuable. It would allow the projects to have a single
>> >> > > view
>> >> > of
>> >> > > provisioning compute, as we can today with Nova and get the
>> >> > benefit of bare
>> >> > > metal through Ironic, VM's through Nova VM's, and containers
>> >> > through
>> >> > > nova-docker.
>> >> > >
>> >> > > With this in place, a project like Trove can offer
>> >> > > database-as-
>> >> a-
>> >> > service on
>> >> > > a spectrum of compute infrastructures as any end-user would
>> >> expect.
>> >> > > Databases don't always make sense in VM's, and while
>> >> > > containers
>> >> > are great
>> >> > > for quick and dirty prototyping, and VM's are great for much
>> >> > > more,
>> >> > there
>> >> > > are databases that will in production only be meaningful on
>> >> bare-
>> >> > metal.
>> >> > >
>> >> > > Therefore, if there is a move towards offering a common API
>> >> > > for
>> >> > VM's,
>> >> > > bare-metal and containers, that would be huge.
>> >> > >
>> >> > > Without such a mechanism, consuming containers in Trove adds
>> >> > considerable
>> >> > > complexity and leads to a very sub-optimal architecture
>> (IMHO).
>> >> > FWIW, a
>> >> > > working prototype of Trove leveraging Ironic, VM's, and nova-
>> >> > docker to
>> >> > > provision databases is something I worked on a while ago, and
>> >> > > have
>> >> > not
>> >> > > revisited it since then (once the direction appeared to be
>> >> Magnum
>> >> > for
>> >> > > containers).
>> >> > >
>> >> > > With all that said, I don't want to downplay the value in a
>> >> > container
>> >> > > specific API. I'm merely observing that from the perspective
>> >> > > of
>> >> a
>> >> > consumer
>> >> > > of computing services, a common abstraction is incredibly
>> >> valuable.
>> >> > >
>> >> > > Thanks,
>> >> > >
>> >> > > -amrith
>> >> > >
>> >> > >
>> >> > > -----Original Message-----
>> >> > > From: Monty Taylor [mailto:mordred at inaugust.com]
>> >> > > Sent: Monday, April 11, 2016 11:31 AM
>> >> > > To: Allison Randal <allison at lohutok.net>; Davanum
>> Srinivas
>> >> > > <davanum at gmail.com>; foundation at lists.openstack.org
>> >> > > Cc: OpenStack Development Mailing List (not for usage
>> >> > questions)
>> >> > > <openstack-dev at lists.openstack.org>
>> >> > > Subject: Re: [openstack-dev] [OpenStack Foundation]
>> >> > [board][tc][all]
>> >> > > One
>> >> > > Platform - Containers/Bare Metal? (Re: Board of Directors
>> >> > > Meeting)
>> >> > >
>> >> > > On 04/11/2016 09:43 AM, Allison Randal wrote:
>> >> > >
>> >> > > On Wed, Apr 6, 2016 at 1:11 PM, Davanum Srinivas
>> <
>> >> > > davanum at gmail.com>
>> >> > >
>> >> > > wrote:
>> >> > >
>> >> > > Reading unofficial notes [1], i found one
>> >> > > topic
>> >> > very
>> >> > > interesting:
>> >> > > One Platform - How do we truly support
>> >> containers
>> >> > and bare
>> >> > > metal
>> >> > > under a common API with VMs? (Ironic, Nova,
>> >> > adjacent
>> >> > > communities e.g.
>> >> > > Kubernetes, Apache Mesos etc)
>> >> > >
>> >> > > Anyone present at the meeting, please expand
>> >> > > on
>> >> > those few
>> >> > > notes on
>> >> > > etherpad? And how if any this feedback is
>> >> getting
>> >> > back to
>> >> > > the
>> >> > > projects?
>> >> > >
>> >> > >
>> >> > > It was really two separate conversations that got
>> >> > conflated in the
>> >> > > summary. One conversation was just being supportive
>> of
>> >> > bare metal,
>> >> > > VMs, and containers within the OpenStack umbrella.
>> The
>> >> > other
>> >> > > conversation started with Monty talking about his
>> work
>> >> on
>> >> > shade,
>> >> > > and
>> >> > > how it wouldn't exist if more APIs were focused on
>> the
>> >> > > way
>> >> > users
>> >> > > consume the APIs, and less an expression of the
>> >> > implementation
>> >> > > details
>> >> > >
>> >> > > of each project.
>> >> > >
>> >> > > OpenStackClient was mentioned as a unified CLI for
>> >> > OpenStack
>> >> > > focused
>> >> > > more on the way users consume the CLI. (OpenStackSDK
>> >> > wasn't
>> >> > > mentioned,
>> >> > > but falls in the same general category of work.)
>> >> > >
>> >> > > i.e. There wasn't anything new in the conversation,
>> it
>> >> > > was
>> >> > more a
>> >> > > matter of the developers/TC members on the board
>> sharing
>> >> > > information
>> >> > > about work that's already happening.
>> >> > >
>> >> > >
>> >> > > I agree with that - but would like to clarify the 'bare
>> >> > > metal,
>> >> > VMs and
>> >> > > containers' part a bit. (an in fact, I was concerned in
>> >> > > the
>> >> > meeting
>> >> > > that
>> >> > > the messaging around this would be confusing because we
>> >> > 'supporting
>> >> > > bare
>> >> > > metal' and 'supporting containers' mean two different
>> >> > > things
>> >> > but we use
>> >> > > one phrase to talk about it.
>> >> > >
>> >> > > It's abundantly clear at the strategic level that having
>> >> > OpenStack be
>> >> > > able
>> >> > > to provide both VMs and Bare Metal as two different sorts
>> >> > > of
>> >> > resources
>> >> > > (ostensibly but not prescriptively via nova) is one of
>> our
>> >> > advantages.
>> >> > > We
>> >> > > wanted to underscore how important it is to be able to do
>> >> > > that,
>> >> > and
>> >> > > wanted
>> >> > > to underscore that so that it's really clear how
>> important
>> >> it
>> >> > is any
>> >> > > time
>> >> > > the "but cloud should just be VMs" sentiment arises.
>> >> > >
>> >> > > The way we discussed "supporting containers" was quite
>> >> > different and
>> >> > > was
>> >> > > not about nova providing containers. Rather, it was about
>> >> > reaching out
>> >> > > to
>> >> > > our friends in other communities and working with them on
>> >> > making
>> >> > > OpenStack
>> >> > > the best place to run things like kubernetes or docker
>> swarm.
>> >> > > Those are systems that ultimately need to run, and it
>> >> > > seems
>> >> > that good
>> >> > > integration (like kuryr with libnetwork) can provide a
>> >> really
>> >> > strong
>> >> > > story. I think pretty much everyone agrees that there is
>> >> > > not
>> >> > much value
>> >> > > to
>> >> > > us or the world for us to compete with kubernetes or
>> docker.
>> >> > >
>> >> > > So, we do want to be supportive of bare metal and
>> >> > > containers
>> >> > > -
>> >> > but the
>> >> > > specific _WAY_ we want to be supportive of those things
>> is
>> >> > different
>> >> > > for
>> >> > > each one.
>> >> > >
>> >> > > Monty
>> >> > >
>> >> > >
>> >> > >
>> >> >
>> >>
>> _____________________________________________________________________
>> >> _
>> >> > _
>> >> > ___
>> >> > > OpenStack Development Mailing List (not for usage
>> questions)
>> >> > > Unsubscribe: OpenStack-dev-request at lists.openstack.org?
>> >> > > subject:unsubscribe
>> >> > >
>> >> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-
>> dev
>> >> > >
>> >> > >
>> >> >
>> >>
>> _____________________________________________________________________
>> >> _
>> >> > _
>> >> > ___
>> >> > > OpenStack Development Mailing List (not for usage questions)
>> >> > > Unsubscribe: OpenStack-dev-
>> >> > request at lists.openstack.org?subject:unsubscribe
>> >> > > http://lists.openstack.org/cgi-
>> bin/mailman/listinfo/openstack-
>> >> dev
>> >> > >
>> >> > >
>> >> >
>> >> > >_________________________________________________________________
>> _
>> >> > >__
>> >> _
>> >> > >_
>> >> > _
>> >> > >___ OpenStack Development Mailing List (not for usage questions)
>> >> > >Unsubscribe:
>> >> > >OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >> >
>> >> >
>> >> > --
>> >> > @flaper87
>> >> > Flavio Percoco
>> >>
>> _____________________________________________________________________
>> >> __
>> >> ___
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe: OpenStack-dev-
>> >> request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>
>> _____________________________________________________________________
>> >> __
>> >> ___
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe: OpenStack-dev-
>> >> request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >______________________________________________________________________
>> _
>> >___ OpenStack Development Mailing List (not for usage questions)
>> >Unsubscribe:
>> >OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________________________________
>> ___
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-
>> request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list