<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Sep 14, 2016 at 11:22 AM, Liping Mao (limao) <span dir="ltr"><<a href="mailto:limao@cisco.com" target="_blank">limao@cisco.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif"><span class="">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">You have a valid point regarding ipvlan support in newer kernel versions but IIUC overlay mode might not help if nic has a limit on max number of macs that it supports in hardware.</div>
</div>
</div>
</blockquote>
<div> for example: <a href="http://www.brocade.com/content/html/en/configuration-guide/fastiron-08030b-securityguide/GUID-ED71C989-6295-4175-8CFE-7EABDEE83E1F.html" target="_blank">http://www.brocade.<wbr>com/content/html/en/<wbr>configuration-guide/fastiron-<wbr>08030b-securityguide/GUID-<wbr>ED71C989-6295-4175-8CFE-<wbr>7EABDEE83E1F.html </a></div>
</div>
</span><div>Thanks vikas point out this. Yes, It may cause problem if the mac of containers expose to hardware switch. </div>
<div>In overlay case, AFAIK, hw should not learn container mac as it is in vxlan(gre) encapsulation.</div></div></blockquote><div><br></div><div>gotcha, thanks Liping.</div><div><br></div><div>What is your opinion on the unicast macs limit that some drivers impose which can enable promiscous mode on the vm if macvlan interfaces cross a certain limit and thus may result into performance degradation by accepting all the multicast/broadcast traffic within subnet ? <br></div><div><br></div><div>ipvlan has problems with dhcp and ipv6. I think its a topic worth discussing.</div><div><br></div><div>-Vikas </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><br>
</div>
<div><br>
</div>
<div>Regards,</div>
<div>Liping Mao</div>
<div><br>
</div>
<span>
<div style="font-family:Calibri;font-size:11pt;text-align:left;color:black;BORDER-BOTTOM:medium none;BORDER-LEFT:medium none;PADDING-BOTTOM:0in;PADDING-LEFT:0in;PADDING-RIGHT:0in;BORDER-TOP:#b5c4df 1pt solid;BORDER-RIGHT:medium none;PADDING-TOP:3pt"><span class="">
<span style="font-weight:bold">From: </span>Vikas Choudhary <<a href="mailto:choudharyvikas16@gmail.com" target="_blank">choudharyvikas16@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
</span><span style="font-weight:bold">Date: </span>2016年9月14日 星期三 下午1:10<div><div class="h5"><br>
<span style="font-weight:bold">To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.<wbr>openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Kuryr] IPVLAN data path proposal<br>
</div></div></div><div><div class="h5">
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Sep 14, 2016 at 10:33 AM, Vikas Choudhary <span dir="ltr">
<<a href="mailto:choudharyvikas16@gmail.com" target="_blank">choudharyvikas16@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote"><span>On Wed, Sep 14, 2016 at 9:39 AM, Liping Mao (limao)
<span dir="ltr"><<a href="mailto:limao@cisco.com" target="_blank">limao@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:calibri,sans-serif">
<span>
<div>> Though, not the best person to comment on macvlan vs ipvlan, one limitation of macvlan is that on physical interfaces, maximum possible number of random mac generations may not cope-up with large number of containers on same vm. </div>
<div><br>
</div>
</span>
<div>Thanks, yes, it is a limitation, Vikas. </div>
<div>This happened if you use vlan as tenant network. If tenant network use overlay mode, maybe it will be a little bit better for the mac problem.</div>
<div>The reason why I mention macvlan can be one of choice is because ipvlan need a very new kernel , it maybe a little bit hard to use in prod env(AFAIK). </div>
</div>
</blockquote>
<div><br>
</div>
</span>
<div>You have a valid point regarding ipvlan support in newer kernel versions but IIUC overlay mode might not help if nic has a limit on max number of macs that it supports in hardware.
</div>
</div>
</div>
</div>
</blockquote>
<div> for example: <a href="http://www.brocade.com/content/html/en/configuration-guide/fastiron-08030b-securityguide/GUID-ED71C989-6295-4175-8CFE-7EABDEE83E1F.html" target="_blank">
http://www.brocade.com/<wbr>content/html/en/configuration-<wbr>guide/fastiron-08030b-<wbr>securityguide/GUID-ED71C989-<wbr>6295-4175-8CFE-7EABDEE83E1F.<wbr>html </a></div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
</div>
</div>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div>
<div>
<div><br>
</div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:calibri,sans-serif">
<div><br>
</div>
<div>Regards,</div>
<div>Liping Mao</div>
<div><br>
</div>
<span>
<div style="font-family:calibri;font-size:11pt;text-align:left;color:black;border-width:1pt medium medium;border-style:solid none none;border-bottom-color:initial;border-left-color:initial;padding:3pt 0in 0in;border-top-color:rgb(181,196,223);border-right-color:initial">
<span style="font-weight:bold">From: </span>Vikas Choudhary <<a href="mailto:choudharyvikas16@gmail.com" target="_blank">choudharyvikas16@gmail.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a>><br>
<span style="font-weight:bold">Date: </span>2016年9月14日 星期三 上午11:50
<div>
<div><br>
<span style="font-weight:bold">To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Kuryr] IPVLAN data path proposal<br>
</div>
</div>
</div>
<div>
<div>
<div><br>
</div>
<div>
<div>
<div dir="ltr"><br>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Wed, Sep 14, 2016 at 7:10 AM, Liping Mao (limao) <span dir="ltr">
<<a href="mailto:limao@cisco.com" target="_blank">limao@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:calibri,sans-serif">
<div>Hi Ivan and Gary,</div>
<div><br>
</div>
<div>maybe we can use macvlan as ipvlan need very new kernel.</div>
<div>allow-address-pairs can aslo allow different mac in vm.</div>
<div>Do we consider macvlan here? Thanks.</div>
</div>
</blockquote>
<div> </div>
<div>Though, not the best person to comment on macvlan vs ipvlan, one limitation of macvlan is that on physical interfaces, maximum possible number of random mac generations may not cope-up with large number of containers on same vm. </div>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:calibri,sans-serif">
<div><br>
</div>
<div>Regards,</div>
<div>Liping Mao</div>
<div><br>
</div>
<span>
<div style="font-family:calibri;font-size:11pt;text-align:left;color:black;border-width:1pt medium medium;border-style:solid none none;border-bottom-color:initial;border-left-color:initial;padding:3pt 0in 0in;border-top-color:rgb(181,196,223);border-right-color:initial">
<span style="font-weight:bold">From: </span>Liping Mao <<a href="mailto:limao@cisco.com" target="_blank">limao@cisco.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a>><br>
<span style="font-weight:bold">Date: </span>2016年9月13日 星期二 下午9:09<br>
<span style="font-weight:bold">To: </span>OpenStack List <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a>>
<div>
<div><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Kuryr] IPVLAN data path proposal<br>
</div>
</div>
</div>
<div>
<div>
<div><br>
</div>
<div>
<div dir="auto">
<div>Hi Gary，</div>
<div><br>
</div>
<div>I mean maybe that can be one choice in my mind.</div>
<div><br>
</div>
<div>Security Group is for each neutron port，in this case，all the docker on one vm will share one neutron port（if I understand correct），then they will share the security group on that port，it is not per container per security group，not sure how to use security
group in this case？ </div>
<div><br>
</div>
<div>Regards，</div>
<div>Liping Mao</div>
<div><br>
在 2016年9月13日，20:31，Loughnane, Gary <<a href="mailto:gary.loughnane@intel.com" target="_blank">gary.loughnane@intel.com</a>> 写道：<br>
<br>
</div>
<blockquote type="cite">
<div>
<div>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Hi Liping,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Thank you for the feedback!<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Do you mean to have disabled security groups as an optional configuration for Kuryr?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Do you have any opinion on the consequences/acceptability of disabling SG?<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Regards,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="color:rgb(31,73,125)">Gary<u></u><u></u></span></p>
<p class="MsoNormal"><a name="m_-9121052605763953698_m_-1006072062604774476_m_-2500911203574663391_m_-359933913728608186__MailEndCompose"><span style="color:rgb(31,73,125)"><u></u> <u></u></span></a></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><a name="m_-9121052605763953698_m_-1006072062604774476_m_-2500911203574663391_m_-359933913728608186______replyseparator"></a><b><span>From:</span></b><span> Liping Mao (limao) [<a href="mailto:limao@cisco.com" target="_blank">mailto:limao@cisco.com</a>]
<br>
<b>Sent:</b> Tuesday, September 13, 2016 12:56 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions) <<a href="mailto:openstack-dev@lists.openstack.org" target="_blank">openstack-dev@lists.openstack<wbr>.org</a>><br>
<b>Subject:</b> Re: [openstack-dev] [Kuryr] IPVLAN data path proposal<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Hi Ivan<span style="font-family:simsun">，</span><span style="font-size:12pt"><u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">It sounds cool<span style="font-family:simsun">！</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">for security group and allowed address pair<span style="font-family:simsun">，</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Maybe we can disable port-security<span style="font-family:simsun">，</span>because all the docker in one vm will share one security group on the vm port. I'm not sure how to use sg for each docker<span style="font-family:simsun">，</span>maybe
just disable port-security can be one of the choice. then do not need allowed address pairs in this case. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Regards<span style="font-family:simsun">，</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Lipimg Mao<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt"><br>
<span style="font-family:simsun">在</span> 2016<span style="font-family:simsun">年</span>9<span style="font-family:simsun">月</span>12<span style="font-family:simsun">日，</span>19:31<span style="font-family:simsun">，</span>Coughlan, Ivan <<a href="mailto:ivan.coughlan@intel.com" target="_blank">ivan.coughlan@intel.com</a>>
<span style="font-family:simsun">写道：</span><u></u><u></u></p>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>Overview</b><u></u><u></u></p>
<p class="MsoNormal">Kuryr proposes to address the issues of double encapsulation and exposure of containers as neutron entities when containers are running within VMs.<u></u><u></u></p>
<p class="MsoNormal">As an alternative to the vlan-aware-vms and use of ovs within the VM, we propose to:<u></u><u></u></p>
<p><span>-<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span>Use allowed-address-pairs configuration for the VM neutron port<u></u><u></u></p>
<p><span>-<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span>Use IPVLAN for wiring the Containers within VM<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">In this way:<u></u><u></u></p>
<p><span>-<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span>Achieve efficient data path to container within VM<u></u><u></u></p>
<p><span>-<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span>Better leverage OpenStack EPA(Enhanced Platform Awareness) features to accelerate the data path (more details below)<u></u><u></u></p>
<p><span>-<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span>Mitigate the risk of vlan-aware-vms not making neutron in time<u></u><u></u></p>
<p><span>-<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span>Provide a solution that works on existing and previous openstack releases<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">This work should be done in a way permitting the user to optionally select this feature.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<h2><b><span style="font-size:11pt;line-height:105%;font-family:calibri,sans-serif;color:windowtext">Required Changes</span></b><u></u><u></u></h2>
<p class="MsoNormal">The four main changes we have identified in the current kuryr codebase are as follows:<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:8pt;margin-left:18pt;line-height:105%"><span><span>·<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span></span>Introduce an option of enabling “IPVLAN in VM” use case. This can be achieved by using a config file option or possibly passing a command line argument. The IPVLAN master interface must also be identified.<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:8pt;margin-left:18pt;line-height:105%"><span><span>·<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span></span>If using “IPVLAN in VM” use case, Kuryr should no longer create a new port in Neutron or the associated VEth pairs. Instead, Kuryr will create a new IPVLAN slave interface on top of the VM’s master interface and pass this slave interface
to the Container netns.<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:8pt;margin-left:18pt;line-height:105%"><span><span>·<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span></span>If using “IPVLAN in VM” use case, the VM’s port ID needs to be identified so we can associate the additional IPVLAN addresses with the port. This can be achieved by querying Neutron’s show-port function and passing the VMs IP address.<u></u><u></u></p>
<p style="margin-right:0cm;margin-bottom:8pt;margin-left:18pt;line-height:105%"><span><span>·<span style="font-style:normal;font-weight:normal;font-size:7pt;line-height:normal;font-family:'times new roman'">
</span></span></span>If using “IPVLAN in VM” use case, Kuryr should associate the additional IPVLAN addresses with the VMs port. This can be achieved using Neutron’s
<span style="font-family:'courier new'">allowed-address-pairs</span> flag in the
<span style="font-family:'courier new'">port-update</span> function. We intend to make use of Kuryr’s existing IPAM functionality to request these IPs from Neutron.<u></u><u></u></p>
<p class="MsoNormal"><b> </b><u></u><u></u></p>
<p class="MsoNormal"><b>Asks</b><u></u><u></u></p>
<p class="MsoNormal">We wish to discuss the pros and cons.<u></u><u></u></p>
<p class="MsoNormal">For example, containers exposure as proper neutron entities and the utility of neutron’s allowed-address-pairs is not yet well understood.<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">We also wish to understand if this approach is acceptable for kuryr?<u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"><b>EPA</b><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(37,37,37);background-image:initial;background-color:white;background-position:initial;background-repeat:initial">The Enhanced Platform Awareness
initiative is a continuous program to enable fine-tuning of the platform for virtualized network functions.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(37,37,37);background-image:initial;background-color:white;background-position:initial;background-repeat:initial">This is done by exposing the
processor and platform capabilities through the management and orchestration layers.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:rgb(37,37,37);background-image:initial;background-color:white;background-position:initial;background-repeat:initial">When a virtual network function
is instantiated by an Enhanced Platform Awareness enabled orchestrator, the application requirements can be more efficiently matched with the platform capabilities.</span><u></u><u></u></p>
<p class="MsoNormal"><a href="http://itpeernetwork.intel.com/openstack-kilo-release-is-shaping-up-to-be-a-milestone-for-enhanced-platform-awareness/" target="_blank">http://itpeernetwork.intel.com<wbr>/openstack-kilo-release-is-sha<wbr>ping-up-to-be-a-milestone-for-<wbr>enhanced-platform-awareness/</a><u></u><u></u></p>
<p class="MsoNormal"><a href="https://networkbuilders.intel.com/docs/OpenStack_EPA.pdf" target="_blank">https://networkbuilders.intel.<wbr>com/docs/OpenStack_EPA.pdf</a><u></u><u></u></p>
<p class="MsoNormal"><a href="https://www.brighttalk.com/webcast/12229/181563/epa-features-in-openstack-kilo" target="_blank">https://www.brighttalk.com/web<wbr>cast/12229/181563/epa-features<wbr>-in-openstack-kilo</a><u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal">Regards,<u></u><u></u></p>
<p class="MsoNormal">Ivan….<u></u><u></u></p>
<p>------------------------------<wbr>------------------------------<wbr>--<br>
Intel Research and Development Ireland Limited<br>
Registered in Ireland<br>
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare<br>
Registered Number: 308263<u></u><u></u></p>
<p>This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all
copies.<u></u><u></u></p>
</div>
</blockquote>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<p class="MsoNormal"><span style="font-size:12pt;font-family:simsun">______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org</a>?subject:unsubscrib<wbr>e<br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><u></u><u></u></span></p>
</div>
</blockquote>
</div>
<p>------------------------------<wbr>------------------------------<wbr>--<br>
Intel Research and Development Ireland Limited<br>
Registered in Ireland<br>
Registered Office: Collinstown Industrial Park, Leixlip, County Kildare<br>
Registered Number: 308263</p>
<p>This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all
copies.</p>
<p></p>
</div>
</blockquote>
<blockquote type="cite">
<div><span>______________________________<wbr>______________________________<wbr>______________</span><br>
<span>OpenStack Development Mailing List (not for usage questions)</span><br>
<span>Unsubscribe: <a href="mailto:OpenStack-dev-request@lists.openstack.org" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org</a>?subject:unsubscrib<wbr>e</span><br>
<span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a></span><br>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</span></div>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</blockquote>
</div>
</div>
</div>
<br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</div></div></span>
</div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div></div>