<html>
<head>
<meta content="text/html; charset=ISO-8859-1"
http-equiv="Content-Type">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Stephen,<br>
Commenting in line below<br>
<br>
<div class="moz-cite-prefix">On 04/16/2014 07:56 PM, Stephen
Balukoff wrote:<br>
</div>
<blockquote
cite="mid:CAAGw+ZpB1-z+Z016nfmAbztmhY9fqSTnDfC4hH5nhdtBdD58Cg@mail.gmail.com"
type="cite">
<meta http-equiv="Content-Type" content="text/html;
charset=ISO-8859-1">
<div dir="ltr">Hi y'all!
<div><br>
</div>
<div>This is actually a pretty good start for a revision of the
Neutron LBaaS API.</div>
<div><br>
</div>
<div>My feedback on your proposed API v2.0 is actually pretty
close to Eugene's, with a couple additions:</div>
<div><br>
</div>
<div>You say 'only one port and protocol per load balancer', yet
I don't know how this works. Could you define what a 'load
balancer' is in this case? (port and protocol are attributes
that I would associate with a TCP or UDP listener of some
kind.) Are you using 'load balancer' to mean 'listener' in
this case (contrary to previous discussion of this on this
list and the one defined here <a moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Neutron/LBaaS/Glossary#Loadbalancer"
target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/Glossary#Loadbalancer</a>
)?<br>
</div>
</div>
</blockquote>
<br>
Yes, it could be considered as a Listener according to that
documentation. The way to have a "listener" using the same VIP but
listen on two different ports is something we call VIP sharing. You
would assign a VIP to one load balancer that uses one port, and then
assign that same VIP to another load balancer but that load balancer
is using a different port than the first one. How the backend
implements it is an implementation detail (redudant, I know). In
the case of HaProxy it would just add the second port to the same
config that the first load balancer was using. In other drivers it
might be different.<br>
<br>
<blockquote
cite="mid:CAAGw+ZpB1-z+Z016nfmAbztmhY9fqSTnDfC4hH5nhdtBdD58Cg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
</div>
<div><br>
</div>
<div>
<div>As pointed out, one pool per load balancer breaks any L7
switching functionality. SSL and L7 were the two major
features that spawned this whole discussion about LBaaS a
couple months ago, so any solution we propose should
probably have these features.</div>
</div>
</div>
</blockquote>
Yes we agree one pool per load balancer breaks L7 switching
functionality. However, as I told Eugene, we also came up with a
"content_switching" object that would be a part of the load balancer
root object. In that object it does define multiple pools and
rules. The details of the pools and rules may indeed need some
tweaking, but that doesn't mean this solution breaks the L7
switching requirement.<br>
<br>
As for SSL, this absolutely allows SSL. Using the common use case
for SSL Termination:<br>
1. Create an HTTP load balancer listening on port 80.<br>
2. Create an HTTPS load balancer listening on port 443 sharing the
same VIP and pool as the first load balancer. Also, add an SSL
Termination/SSL Decryption object to this 2nd load balancer. <br>
<br>
We did not say much about the SSL Termination/SSL Decryption object
because we wanted to make sure it was able to meet other
requirements before we started to discuss that.<br>
<blockquote
cite="mid:CAAGw+ZpB1-z+Z016nfmAbztmhY9fqSTnDfC4hH5nhdtBdD58Cg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
<div><br>
</div>
<div>Context switching is the *only* reason to have multiple
pools per load balancer... and I really just don't
understand where the "consistency" argument between having
"a pool" vs. "pools." I don't understand why one would think
having multiple pools for a load balancer (that doesn't need
them) would be a desired way to handle this "inconsistency"
problem. Anyway... There's been discussion of this
previously here: <a moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Neutron/LBaaS/l7"
target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/l7</a>
...and I think I can illustrate (via proposed API) a better
way to do this... (in a nutshell, you need to have an
additional object which links listeners to pools via a
policy or rule. API is going to need to have controls to
modify these rules.)</div>
</div>
<div><br>
</div>
<div>I'm not sure I fully understand the requirements behind the
"single API call" proposal for creating a LBaaS service
instance (whatever that means). Therefore, for now, I'm going
to withhold any judgement on this or anything attempting to
meet this requirement. Where does this need come from, and
what are people expecting to see for their "single API call"?</div>
</div>
</blockquote>
The "single API call" is something we do currently use. One reason
to have it is because it is easier to understand from a user
standpoint that creating a fully provisioned load balancer is done
in one step at the /loadbalancers resource instead of having to go
to 3 or more additional resources to do this. Now, since 90% of
users will most likely be using some kind of GUI to do this it might
minimize the need for this, but we still feel that creating less
confusion is best.<br>
<br>
Another reason we prefer this is because we have experience in an
API that does need to make many calls and steps before a load
balancer is actually up and running. Once the environment all of
the load balancers lived in became more and more dense, the
provisioning time increased by many folds. Once we were able to use
an API that used the same backend and environment but every step was
done in one call, the provisioning time was cut down by a factor of
20. Now this may just be a fluke only we encounter but I don't see
how having a single API create call hurts anything. The CLI client
can still only have options to create a load balancer in 3 or more
steps.<br>
<blockquote
cite="mid:CAAGw+ZpB1-z+Z016nfmAbztmhY9fqSTnDfC4hH5nhdtBdD58Cg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div><br>
</div>
<div>I'd like to take a stab at revising this API to reflect
both the terminology defined in the glossary here: <a
moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Neutron/LBaaS/Glossary"
target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/Glossary</a>
as well as addressing features having to do with SSL, L7 and
(if y'all will let me) HA. I would also work off the
requirements documents here:</div>
<div><a moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements"
target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements</a><br>
</div>
<div>Features wishlist here:</div>
<div><a moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Neutron/LBaaS/Usecases">https://wiki.openstack.org/wiki/Neutron/LBaaS/Usecases</a><br>
</div>
<div>Moderated by the real-world feature usage here:</div>
<div><a moz-do-not-send="true"
href="https://docs.google.com/spreadsheet/ccc?key=0Ar1FuMFYRhgadDVXZ25NM2NfbGtLTkR0TDFNUWJQUWc&usp=sharing"
target="_blank">https://docs.google.com/spreadsheet/ccc?key=0Ar1FuMFYRhgadDVXZ25NM2NfbGtLTkR0TDFNUWJQUWc&usp=sharing</a><br>
</div>
<div>... to try to create an API which addresses as much of this
as possible (with appropriate object model diagrams for
reference), yet still has "sane defaults" for simple use
cases.</div>
<div><br>
</div>
<div>As an aside, it seems everyone's number one feature request
at this time is HA. (more so than SSL and L7, yo!)</div>
<div><br>
</div>
<div>Note that I certainly won't have this ready for tomorrow's
meeting, but could probably have a draft to show y'all at next
week's meeting if y'all think it would be helpful to produce
such a thing. Anyway, we can discuss this at tomorrow's
meeting...<br>
</div>
</div>
</blockquote>
Please take a stab at it, the more ideas we all see the better we
can make the revised API. We can incorporate all the good ideas
into one great API. At least that is the hope<br>
<blockquote
cite="mid:CAAGw+ZpB1-z+Z016nfmAbztmhY9fqSTnDfC4hH5nhdtBdD58Cg@mail.gmail.com"
type="cite">
<div dir="ltr">
<div>
</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Stephen</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Wed, Apr 16, 2014 at 4:17 PM, Carlos
Garza <span dir="ltr"><<a moz-do-not-send="true"
href="mailto:carlos.garza@rackspace.com" target="_blank">carlos.garza@rackspace.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0
.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word">
<br>
<div>
<div class="">
<div>On Apr 16, 2014, at 4:31 PM, Eugene Nikanorov
<<a moz-do-not-send="true"
href="mailto:enikanorov@mirantis.com"
target="_blank">enikanorov@mirantis.com</a>>
wrote:</div>
<br>
<blockquote type="cite">
<div dir="ltr">Hi folks,
<div><br>
</div>
<div>I've briefly looked over the doc.</div>
<div><br>
</div>
<div>I think whole idea to base the API on Atlas
misses the content switching use case, which is
very important:</div>
<div>We need multiple pools within loadbalancer,
and API doesn't seem to allow that.</div>
<div>If it would, then you'll face another
problem: you need to reference those pools
somehow inside the json you use in POST.</div>
<div>There are two options here: use names or IDs,
both are putting constraints and create
complexity for both user of such API and for the
implementation.</div>
<div><br>
</div>
<div>That particular problem becomes worse when it
comes to objects which might not have names
while it's better to not provide ID in POST and
rely on their random generation. E.g. when you
need to create references between objects in
json input - you'll need to create artificial
attributes just for the parser to understand
that such input means.</div>
<div><br>
</div>
<div>So that makes me think that right now a
'single-call API' is not flexible enough to
comply with our requirements.</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div> We have demonstrated that you can create
loadbalancers in separate transactions and in a single
call fashion using both reference_ids to previous
pools and as well as using a transient names to create
objects in the same single call and reference them
later on in other objects. The single call API is very
flexible in that it allows you to create sub
objects(We proposed transient ids to allow the user to
avoid creating duplicate objects with different ids)
on the fly as well as reference preexisting objects by
id. The allowance for transient ids is adding
flexibility to the api not taking away from it as you
declared. I would like you to really be clear on what
"our requirements"? What requirement is our single API
call violating?</div>
<div><br>
</div>
<div> We have thus far attempted to support a single
call API that doesn't interfere with multiple smaller
object creation calls. If we are just adding to the
API in a demonstrably workable fashion what is the
real objection.</div>
<div class="">
<div><br>
</div>
<br>
<blockquote type="cite">
<div dir="ltr">
<div>While I understand that it might be simpler
to use such API for some cases, it makes complex
configurations fall back to our existing
approach which is creating configuration on per
object basis.</div>
<div>While the problem with complex configurations
is not sorted out, I'd prefer if we focus on
existing 'object-oriented' approach.</div>
</div>
</blockquote>
<div><br>
</div>
</div>
<div> Your basically saying</div>
<div>P1: "The single API call proposal doesn't support
*ALL* complex configurations"</div>
<div>P2: " if the single API proposal doesn't support
*ALL* complex configurations the proposal should be
rejected"</div>
<div><br>
</div>
<div>We have demonstrated that the proposed single API
call can handle complex configurations via transient
ids. So we already disagree with preposition 1.</div>
<div><br>
</div>
<div>We don't agree with preposition 2 either: </div>
<div> We believe it is unfair to punish the API end
user due to the religious belief that "The single API
calls must support all possible configurations or you
as the customer can't be allowed to use the single API
call even for simpler configurations."</div>
<div><br>
</div>
<div>We want the single API call proposal to be as
useful as possible so we are like wise looking at ways
to have it solve ALL complex configurations and so far
we feel transient IDs solve this problem already. </div>
<div><br>
</div>
<div> Is the real objection that a single API call
makes the implementation too complex? We are
advocating that a single API makes it easier on the
end user of the API and are of the impression that its
better to have a complex implementation inside our
neutron/lbaas code rather then passing that complexity
down to the end user of the API.</div>
<div><br>
</div>
<div> We don't object to multiple smaller object
creation transactions we just want the addition of
having single API call.</div>
<div class="">
<div><br>
</div>
<div><br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>On the other hand, without single-call API
the rest of proposal seems to be similar to
approaches discussed in <a
moz-do-not-send="true"
href="https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance/Discussion"
target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance/Discussion</a></div>
</div>
</blockquote>
</div>
<div> Since you linked the object model proposals
could you also link the "rest of the proposals" or are
you referring to our draft as "rest of proposal"?</div>
<div>
<div class="h5">
<div><br>
</div>
<div><br>
</div>
<blockquote type="cite">
<div dir="ltr">
<div>Thanks,</div>
<div>Eugene.</div>
<div><br>
</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Thu, Apr 17, 2014 at
12:59 AM, Brandon Logan <span dir="ltr">
<<a moz-do-not-send="true"
href="mailto:brandon.logan@rackspace.com"
target="_blank">brandon.logan@rackspace.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote"
style="margin:0 0 0 .8ex;border-left:1px
#ccc solid;padding-left:1ex">
<div>
<div
style="direction:ltr;font-size:10pt;font-family:Tahoma">Sorry
about that. It should be readable now.<br>
<div
style="font-size:16px;font-family:Times
New Roman">
<hr>
<div style="direction:ltr"><font
face="Tahoma"><b>From:</b> Eugene
Nikanorov [<a
moz-do-not-send="true"
href="mailto:enikanorov@mirantis.com"
target="_blank">enikanorov@mirantis.com</a>]<br>
<b>Sent:</b> Wednesday, April 16,
2014 3:51 PM
<div><br>
<b>To:</b> OpenStack Development
Mailing List (not for usage
questions)<br>
</div>
<div>
<div><b>Subject:</b> Re:
[openstack-dev]
[Neutron][LBaaS] Requirements
and API revision progress<br>
</div>
</div>
</font><br>
</div>
<div>
<div>
<div>
<div dir="ltr">Hi Brandon,
<div><br>
</div>
<div>Seems that doc has not
been made public, so please
share.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Eugene.</div>
</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On
Thu, Apr 17, 2014 at 12:39
AM, Brandon Logan <span
dir="ltr">
<<a
moz-do-not-send="true"
href="mailto:brandon.logan@rackspace.com"
target="_blank">brandon.logan@rackspace.com</a>></span>
wrote:<br>
<blockquote
class="gmail_quote"
style="margin:0 0 0
.8ex;border-left:1px #ccc
solid;padding-left:1ex">
<div
style="font-size:14px;font-family:Calibri,sans-serif;word-wrap:break-word">
<div>Here is Jorge and
team’s API proposal
based on Atlas. The
document has some
questions and answers
about why decisions
were made. Feel free
to open up a
discussion about these
questions and answers
and really about
anything. This can
be changed up to fit
any flaws or use cases
we missed that this
would not support.</div>
<div><br>
</div>
<div>There is a CLI
example at the bottom
along with a possible
L7 switching API
model.</div>
<div><br>
</div>
<div><a
moz-do-not-send="true"
href="https://docs.google.com/document/d/1mTfkkdnPAd4tWOMZAdwHEx7IuFZDULjG9bTmWyXe-zo/edit"
target="_blank">https://docs.google.com/document/d/1mTfkkdnPAd4tWOMZAdwHEx7IuFZDULjG9bTmWyXe-zo/edit</a></div>
<div><br>
</div>
<div>Thanks,</div>
<div>Brandon Logan</div>
<div><br>
</div>
<span>
<div
style="border-right:medium
none;padding-right:0in;padding-left:0in;padding-top:3pt;text-align:left;font-size:11pt;border-bottom:medium
none;font-family:Calibri;border-top:#b5c4df
1pt
solid;padding-bottom:0in;border-left:medium
none">
<span
style="font-weight:bold">From:
</span>Eugene
Nikanorov <<a
moz-do-not-send="true"
href="mailto:enikanorov@mirantis.com" target="_blank">enikanorov@mirantis.com</a>><br>
<span
style="font-weight:bold">Reply-To:
</span>"<a
moz-do-not-send="true"
href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>"
<<a
moz-do-not-send="true"
href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>><br>
<span
style="font-weight:bold">Date:
</span>Tuesday,
April 15, 2014 at
7:00 AM<br>
<span
style="font-weight:bold">To:
</span>"<a
moz-do-not-send="true"
href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>"
<<a
moz-do-not-send="true"
href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack.org</a>>
<div><br>
<span
style="font-weight:bold">Subject:
</span>Re:
[openstack-dev]
[Neutron][LBaaS]
Requirements and
API revision
progress<br>
</div>
</div>
<div>
<div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Hi
Stephen,
<div><br>
</div>
<div>Thanks
for a good
summary. Some
comments
inline.</div>
<div
class="gmail_extra"><br>
<br>
<div
class="gmail_quote">On
Tue, Apr 15,
2014 at 5:20
AM, Stephen
Balukoff <span
dir="ltr">
<<a
moz-do-not-send="true"
href="mailto:sbalukoff@bluebox.net" target="_blank">sbalukoff@bluebox.net</a>></span>
wrote:
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>So! On
this front:</div>
<div><br>
</div>
<div>1. Does
is make sense
to keep
filling out
use cases in
Samuel's
document
above? I can
think of
several more
use cases that
our customers
actually use
on our current
deployments
which aren't
considered in
the 8 cases in
Samuel's
document thus
far. Plus
nobody has
create any use
cases from the
cloud operator
perspective
yet.</div>
</div>
</blockquote>
<div><br>
</div>
<div>I treat
Sam's doc as a
source of use
cases to
triage API
proposals. If
you think you
have use cases
that don't fit
into existing
API or into
proposed API,
they should
certainly be
brought to
attention.</div>
<div> </div>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>2. It
looks like
we've started
to get
real-world
data on Load
Balancer
features in
use in the
real world. If
you've not
added your
organization's
data, please
be sure to do
so soon so we
can make
informed
decisions
about product
direction. On
this front,
when will we
be making
these
decisions?</div>
</div>
</blockquote>
<div>I'd say
we have two
kinds of
features - one
kind is
features that
affect or even
define the
object model
and API.</div>
<div>Other
kind are
features that
are
implementable
within
existing/proposed
API or require
slight
changes/evolution.</div>
<div>First
kind is the
priority:
while some of
such features
may or may not
be implemented
in some
particular
release, we
need to
implement
proper
infrastructure
for them (API,
obj model)</div>
<div><br>
</div>
<div>Oleg
Bondarev (he's
neutron core)
and me are
planning and
mostly
interested to
work on
implementing
generic stuff
like API/obj
model and
adopt haproxy
driver to it.
So our goal is
to make
implementation
of particular
features
simpler for
contributors
and also make
sure that
proposed
design fits in
general lbaas
architecture.
I believe that
everyone who
wants to see
certain
feature may
start working
on it -
propose
design,
participate in
discussions
and start
actually
writing the
code.</div>
<div> </div>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>3.
Jorge-- I know
an action item
from the last
meeting was to
draft a
revision of
the API
(probably
starting from
something
similar to the
Atlas API).
Have you had a
chance to get
started on
this, and are
you open for
collaboration
on this
document at
this time?
Alternatively,
I'd be happy
to take a stab
at it this
week (though
I'm not very
familiar with
the Atlas
API-- so my
proposal might
not look all
that similar).</div>
</div>
</blockquote>
<div> </div>
<div>+1, i'd
like to see
something as
well. </div>
<div><br>
</div>
<blockquote
class="gmail_quote"
style="margin:0
0 0
.8ex;border-left:1px
#ccc
solid;padding-left:1ex">
<div dir="ltr">
<div><br>
</div>
<div>What
format or
template
should we be
following to
create the API
documentation?
(I see this
here: <a
moz-do-not-send="true"
href="http://docs.openstack.org/api/openstack-network/2.0/content/ch_preface.html"
target="_blank">http://docs.openstack.org/api/openstack-network/2.0/content/ch_preface.html</a>
but this
seems like it
might be a
little heavy
for an API
draft that is
likely to get
altered
significantly,
especially
given how this
discussion has
gone thus far.
:/ )</div>
</div>
</blockquote>
<div><br>
</div>
<div>Agree,
that's too
heavy for API
sketch. I
think a set of
resources with
some
attributes
plus a few cli
calls is what
could show the
picture.</div>
<div><br>
</div>
<div>Thanks,</div>
<div>Eugene.</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org"
target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote>
</div>
</div>
</div>
<br>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a moz-do-not-send="true"
href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a moz-do-not-send="true"
href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"
target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
<br clear="all">
<div><br>
</div>
-- <br>
<span></span>Stephen Balukoff
<br>
Blue Box Group, LLC
<br>
(800)613-4305 x807
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset>
<br>
<pre wrap="">_______________________________________________
OpenStack-dev mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a>
</pre>
</blockquote>
<br>
</body>
</html>