<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; ">
<div>Responses in-line:</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Vijay B <<a href="mailto:os.vbvs@gmail.com">os.vbvs@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, October 14, 2014 7:08 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Neutron][LBaaS][Octavia] Floating IP management<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">Hi Phillip,</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">Adding my thoughts below. I’ll first answer the questions you raised with what I think should be done, and then give my explanations to reason through with those views.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">1. Do we want to add logic in the plugin to call the FLIP association API?</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>>> We should implement the logic in the new v2 extension and the plugin layer as a single API call. We would need to add to the existing
v2 API to be able to do this. The best place to add this option of passing the FLIP info/request to the VIP is in the VIP create and update API calls via new parameters.</p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>>>>>Agreed, we would need to have another field for the FLIP itself. Although, I was under the impression we would use the associate_<span style="font-style: italic; ">floating_</span>ip and not have to modify the VIP call at all. I may be misunderstanding
where you think this should happen.</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">2. If we have logic in the plugin should we have configuration that identifies whether to use/return the FLIP instead of the port VIP?</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>>> Yes and no, in that we should return the complete result of the VIP create/update/list/show API calls, in which we show the VIP internal
IP, but we also show the FLIP either as empty or having a FLIP uuid. External users will anyway use only the FLIP, else they wouldn’t be able to reach the LB and the VIP IP, but the APIs need to show both fields.</p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>>>>> By this, do you mean we would be altering the vip_port API calls? Again, I was under the impression that we would utilize neutrons FLIP calls. With the question #2 I was referring to 'replacing' the VIP on the LB object with a configuration value
rather than showing both or just the VIP. If by VIP CRUD you are talking about LBaaS API then this makes sense to me and we should indeed show them both.</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">3. Would we rather have logic for FLIP association in the drivers?</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>>> This is the hardest part to decide. To understand this, we need to look at two important drivers of LBaaS design:</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>I) The Neutron core plugin we’re using.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>II) The different types of LB devices - physical, virtual standalone, and virtual controlled by a management plane. This leads to different
kinds of LBaaS drivers and different kinds of interaction or the lack of it between them and the core neutron plugin.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">The reason we need to take into account both these is that port provisioning as well as NATing for the FLIP to internal VIP IP will be configured differently by the different network
management/backend planes that the plugins use, and the way drivers configure LBs can be highly impacted by this.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">For example, we can have an NSX infrastructure that will implement the FLIP to internal IP conversion in the logical router module which sits pretty much outside of Openstack’s realm,
using openflow. Or we can use lighter solutions directly on the hypervisor that still employ open flow entries without actually having a dedicated logical routing module. Neither will matter much if we are in a position to have them deploy our networking for
us, i.e., in the cases of us using virtual LBs sitting on compute nodes. But if we have a physical LB, the neutron plugins cannot do much of the network provisioning work for us, typically because physical LBs usually sit outside of the cloud, and are among
the earliest points of contact from the external world.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">This already nudges us to consider putting the FLIP provisioning functionality in the driver. However, consider again more closely the major ways in which LBaaS drivers talk to LB solutions
today depending on II) :</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>a) LBaaS drivers that talk to a virtual LB device on a compute node, directly.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>b) LBaaS drivers that talk to a physical LB device (or a virtual LB sitting outside the cloud) directly.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>c) LBaaS drivers that talk to a management plane like F5’s BigIQ, or Netscaler’s NCC, or as in our case, Octavia, that try to provide tenant
based provisioning of virtual LBs.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>d) The HAProxy reference namespace driver.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">d) is really a PoC use case, and we can forget it. Let’s consider a), b) and c).</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">If we use a) or b), we must assume that the required routing for the virtual LB has been setup correctly, either already through nova or manually. So we can afford to do our FLIP plumbing
in the neutron plugin layer, but, driven by the driver - how? - typically, after the VIP is successfully created on the LB, and just before the driver updates the VIP’s status as ACTIVE, it can create the FLIP. Of course, if the FLIP provisioning fails for
any reason, the VIP still stands. It’ll be empty in the result, and the API will error out saying “VIP created but FLIP creation failed”. It must be manually deleted by another delete VIP call. We can’t afford to provision a FLIP before a VIP is active, for
external traffic shouldn’t be taken while the VIP isn’t up yet. If the lines are getting hazy right now because of this callback model, let’s just focus on the point that we’re initiating FLIP creation in the driver layer while the code sits in the plugin
layer because it will need to update the database. But in absolute terms, we’re doing it in the driver.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">It is use case c) that is interesting. In this case, we should do all neutron based provisioning neither in the driver, nor in the plugin in neutron, rather, we should do this in Octavia,
and in the Octavia controller to be specific. This is very important to note, because if customers are using this deployment (which today has the potential to be way greater in the near future than any other model simply because of the sheer existing customer
base), we can’t be creating the FLIP in the plugin layer and have the controller reattempt it. Indeed, the controllers can change their code to not attempt this, but like just noted, it isn’t logically orderly to create the FLIP before creating the VIP successfully.
So let’s leave this op to the driver.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">I am guilty of not following the proceedings on the Octavia controller design due to internal work deadlines, but if I remember right, the design specified a controller being built to
spawn and configure virtual HAProxy LB instances. If Octavia is not taking that route today, I think that the best place to do this provisioning would be as described above for a) and b).</p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>>>>>These are good points! Having the FLIP creation in the driver or in Octavias case the controller code that manages the HA of the VLBs is a great idea. Would this mean though that we are bypassing Neutrons calls completely if we do this? Or would this
be API calls made from the controller/driver to Neutron? If were bypassing neutron we should assume that the deployers environment has the plumbing set up for the FLIP pool and some kind of service that can be queried for the FLIP? </div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">4) If logic is in the drivers would we still return the port VIP to the user then later overwrite it with the FLIP?</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>>> As described in 2), we always return the VIP port along with the FLIP info in API calls. Users like PaaS that sit on top of IaaS (Openstack)
will only use the FLIP when they commission endpoints for their users, who will only be concerned with the FLIP. </p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">5) Or would we have configuration to not return the port VIP initially, but an additional query would show the associated FLIP.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>>> Not inclined to doing this :) It’s just another field associated with the VIP - an API call for it would be overkill.</p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>>>>>Makes sense to me</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">6) Is there an internal service call for this, and if so would we use it instead of API calls?</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><span class="" style="white-space:pre"></span>>> Really depends on the core plugin you’re using - each plugin does this differently, but all you need to do is to simply call core_plugin.create_floatingip()
and it should do everything for you - it should correctly create the FLIP entry in the db, then go to its backend (a controller or ovs or the L3 agent etc) which will correctly provision the necessary interfaces and/or program the required flow rules or iptables
rules or equivalent and return success or failure.</p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>>>>> I may need to dig into the docs a bit more, but if theres a core_plugin.associate_flip call as there is in the API then this would save us from writing API communications from the controller/driver if we go that route.</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">All said and done, the fact is that in LBaaS, as with many other components, we can always override the default plugin and driver layers and write our own and do whatever we want in
them :) This allows vendors to rewrite implementations if they need to. Hopefully we’ll get the design well enough that they don’t have to do that in more than a few modules.</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">Hope this helps! Please do let me know if you have any questions or need more pointers with the extensions/neutron/etc.</p>
</div>
</div>
</div>
</div>
</span>
<div><br>
</div>
<div>>>>> Thanks for the response and information! I may have a few points fundamentally incorrect, which is reasoning for this discussion, so I do appreciate the time you've taken to discuss with me :)</div>
<span id="OLK_SRC_BODY_SECTION">
<div>
<div>
<div dir="ltr">
<div>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35);min-height:15px">
<br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">Cheers!</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">Regards,</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)">Vijay</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><br>
</p>
<p style="margin:0px;font-size:13px;font-family:Arial;color:rgb(35,35,35)"><br>
</p>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Oct 13, 2014 at 1:18 PM, Phillip Toohill <span dir="ltr">
<<a href="mailto:phillip.toohill@rackspace.com" target="_blank">phillip.toohill@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Diagrams in jpeg format..<br>
<br>
On 10/12/14 10:06 PM, "Phillip Toohill" <<a href="mailto:phillip.toohill@RACKSPACE.COM">phillip.toohill@RACKSPACE.COM</a>><br>
wrote:<br>
<div class="HOEnZb">
<div class="h5"><br>
>Hello all,<br>
><br>
>Heres some additional diagrams and docs. Not incredibly detailed, but<br>
>should get the point across.<br>
><br>
>Feel free to edit if needed.<br>
><br>
>Once we come to some kind of agreement and understanding I can rewrite<br>
>these more to be thorough and get them in a more official place. Also, I<br>
>understand theres other use cases not shown in the initial docs, so this<br>
>is a good time to collaborate to make this more thought out.<br>
><br>
>Please feel free to ping me with any questions,<br>
><br>
>Thank you<br>
><br>
><br>
>Google DOCS link for FLIP folder:<br>
><a href="https://drive.google.com/folderview?id=0B2r4apUP7uPwU1FWUjJBN0NMbWM&usp=sh" target="_blank">https://drive.google.com/folderview?id=0B2r4apUP7uPwU1FWUjJBN0NMbWM&usp=sh</a><br>
>a<br>
>ring<br>
><br>
>-diagrams are <a href="http://draw.io" target="_blank">draw.io</a> based and can be opened from within Drive by<br>
>selecting the appropriate application.<br>
><br>
>On 10/7/14 2:25 PM, "Brandon Logan" <<a href="mailto:brandon.logan@RACKSPACE.COM">brandon.logan@RACKSPACE.COM</a>> wrote:<br>
><br>
>>I'll add some more info to this as well:<br>
>><br>
>>Neutron LBaaS creates the neutron port for the VIP in the plugin layer<br>
>>before drivers ever have any control. In the case of an async driver,<br>
>>it will then call the driver's create method, and then return to the<br>
>>user the vip info. This means the user will know the VIP before the<br>
>>driver even finishes creating the load balancer.<br>
>><br>
>>So if Octavia is just going to create a floating IP and then associate<br>
>>that floating IP to the neutron port, there is the problem of the user<br>
>>not ever seeing the correct VIP (which would be the floating iP).<br>
>><br>
>>So really, we need to have a very detailed discussion on what the<br>
>>options are for us to get this to work for those of us intending to use<br>
>>floating ips as VIPs while also working for those only requiring a<br>
>>neutron port. I'm pretty sure this will require changing the way V2<br>
>>behaves, but there's more discussion points needed on that. Luckily, V2<br>
>>is in a feature branch and not merged into Neutron master, so we can<br>
>>change it pretty easily. Phil and I will bring this up in the meeting<br>
>>tomorrow, which may lead to a meeting topic in the neutron lbaas<br>
>>meeting.<br>
>><br>
>>Thanks,<br>
>>Brandon<br>
>><br>
>><br>
>>On Mon, 2014-10-06 at 17:40 +0000, Phillip Toohill wrote:<br>
>>> Hello All,<br>
>>><br>
>>> I wanted to start a discussion on floating IP management and ultimately<br>
>>> decide how the LBaaS group wants to handle the association.<br>
>>><br>
>>> There is a need to utilize floating IPs(FLIP) and its API calls to<br>
>>> associate a FLIP to the neutron port that we currently spin up.<br>
>>><br>
>>> See DOCS here:<br>
>>><br>
>>> ><br>
>>><a href="http://docs.openstack.org/api/openstack-network/2.0/content/floatingip_c" target="_blank">http://docs.openstack.org/api/openstack-network/2.0/content/floatingip_c</a><br>
>>>r<br>
>>>eate.html<br>
>>><br>
>>> Currently, LBaaS will make internal service calls (clean interface :/)<br>
>>>to create and attach a Neutron port.<br>
>>> The VIP from this port is added to the Loadbalancer object of the Load<br>
>>>balancer configuration and returned to the user.<br>
>>><br>
>>> This creates a bit of a problem if we want to associate a FLIP with the<br>
>>>port and display the FLIP to the user instead of<br>
>>> the ports VIP because the port is currently created and attached in the<br>
>>>plugin and there is no code anywhere to handle the FLIP<br>
>>> association.<br>
>>><br>
>>> To keep this short and to the point:<br>
>>><br>
>>> We need to discuss where and how we want to handle this association. I<br>
>>>have a few questions to start it off.<br>
>>><br>
>>> Do we want to add logic in the plugin to call the FLIP association API?<br>
>>><br>
>>> If we have logic in the plugin should we have configuration that<br>
>>>identifies weather to use/return the FLIP instead the port VIP?<br>
>>><br>
>>> Would we rather have logic for FLIP association in the drivers?<br>
>>><br>
>>> If logic is in the drivers would we still return the port VIP to the<br>
>>>user then later overwrite it with the FLIP?<br>
>>> Or would we have configuration to not return the port VIP initially,<br>
>>>but an additional query would show the associated FLIP.<br>
>>><br>
>>><br>
>>> Is there an internal service call for this, and if so would we use it<br>
>>>instead of API calls?<br>
>>><br>
>>><br>
>>> Theres plenty of other thoughts and questions to be asked and discussed<br>
>>>in regards to FLIP handling,<br>
>>> hopefully this will get us going. I'm certain I may not be completely<br>
>>>understanding this and<br>
>>> is the hopes of this email to clarify any uncertainties.<br>
>>><br>
>>><br>
>>><br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> OpenStack-dev mailing list<br>
>>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
>>_______________________________________________<br>
>>OpenStack-dev mailing list<br>
>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
><br>
>_______________________________________________<br>
>OpenStack-dev mailing list<br>
><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</div>
</div>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>