<html><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:16px"><div id="yui_3_16_0_1_1433870045885_7566">Hi Daniel,</div><div id="yui_3_16_0_1_1433870045885_7565"><br></div><div id="yui_3_16_0_1_1433870045885_7545"><span>I see following in your command</span></div><div id="yui_3_16_0_1_1433870045885_7642"><br><span></span></div><div id="yui_3_16_0_1_1433870045885_7706" dir="ltr">--dhcp-range=set:tag0,<b id="yui_3_16_0_1_1433870045885_7714">192.168.110.0</b>,static,86400s --dhcp-range=set:tag1,<b>192.168.111.0</b>,static,86400s</div><div id="yui_3_16_0_1_1433870045885_7717" dir="ltr"><br></div><div dir="ltr"><br></div><div id="yui_3_16_0_1_1433870045885_7788" dir="ltr">Is this expected? Was this command generated by the agent itself, or was Dnsmasq manually started?<br></div><div id="yui_3_16_0_1_1433870045885_7715" dir="ltr"><br></div><div id="yui_3_16_0_1_1433870045885_7787" dir="ltr"><br></div><div id="yui_3_16_0_1_1433870045885_7786" dir="ltr"><br></div> <div id="yui_3_16_0_1_1433870045885_7785" style="" class=""><span style="" class=""><br style="" class=""></span></div><br><div class="qtdSeparateBR"><br><br></div><div style="display: block;" class="yahoo_quoted"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div style="font-family: HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif; font-size: 16px;"> <div dir="ltr"> <font face="Arial" size="2"> On Tuesday, June 9, 2015 4:41 AM, Kevin Benton <blak111@gmail.com> wrote:<br> </font> </div> <br><br> <div class="y_msg_container"><div id="yiv8713744027"><div><div dir="ltr">><span style="font-size:12.8000001907349px;">Just to be sure, I assume we're focussing here on the issue that Daniel reported</span><div><span style="font-size:12.8000001907349px;"><br clear="none"></span></div><div><span style="font-size:12.8000001907349px;">Yes.</span></div><div><span style="font-size:12.8000001907349px;"><br clear="none"></span></div><div><span style="font-size:12.8000001907349px;">></span><span style="font-size:12.8000001907349px;">To be clear, though, what code are you trying to reproduce on? Current master?</span></div><span class="yiv8713744027im" style="font-size:12.8000001907349px;"><br clear="none"></span><div>I was trying on 2014.1.3, which is the version I understand to be on Fuel 5.1.1.</div><div><br clear="none"></div><div>><span style="font-size:12.8000001907349px;">I'm not clear whether that would qualify as 'concurrent', in the sense that you have in mind.</span></div><br style="font-size:12.8000001907349px;" clear="none"><div>It doesn't look like it based on the pseudocode. I was thinking of a condition where a port is deleted nearly very quickly after it was created. Is that possible with your test? If not, then my theory about out-of-order notifications might not be any good.</div></div><div class="yiv8713744027gmail_extra"><br clear="none"><div class="yiv8713744027gmail_quote">On Tue, Jun 9, 2015 at 3:34 AM, Neil Jerram <span dir="ltr"><<a rel="nofollow" shape="rect" ymailto="mailto:Neil.Jerram@metaswitch.com" target="_blank" href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a>></span> wrote:<br clear="none"><blockquote class="yiv8713744027gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><span class="yiv8713744027">On 09/06/15 01:15, Kevin Benton wrote:<br clear="none">
</span><blockquote class="yiv8713744027gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I'm having difficulty reproducing the issue. The bug that Neil<br clear="none">
referenced (<a rel="nofollow" shape="rect" target="_blank" href="https://bugs.launchpad.net/neutron/+bug/1192381">https://bugs.launchpad.net/neutron/+bug/1192381</a>) looks like<br clear="none">
it was in Icehouse well before the 2014.1.3 release that looks like Fuel<br clear="none">
5.1.1 is using.<br clear="none">
</blockquote>
<br clear="none">
Just to be sure, I assume we're focussing here on the issue that Daniel reported (IP appears twice in Dnsmasq config), and for which I described a possible corollary (Dnsmasq config size keeps growing), and NOT on the "Another DHCP agent problem" that I mentioned below. :-)<br clear="none">
<br clear="none">
BTW, now that I've reviewed the history of when my team saw this, I can say that it was actually first reported to us with the 'IP appears twice in Dnsmasq config' symptom - i.e. exactly the same as Daniel's case. The fact of the Dnsmasq config increasing in size was noticed later.<span class="yiv8713744027"><br clear="none">
<br clear="none">
</span><blockquote class="yiv8713744027gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
I tried setting the agent report interval to something higher than the<br clear="none">
downtime to make it seem like the agent is failing sporadically to the<br clear="none">
server, but it's not impacting the notifications.<br clear="none">
</blockquote>
<br clear="none">
Makes sense - that's the effect of the fix for 1192381.<br clear="none">
<br clear="none">
To be clear, though, what code are you trying to reproduce on? Current master?<span class="yiv8713744027"><br clear="none">
<br clear="none">
</span><blockquote class="yiv8713744027gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Neil, does your testing where you saw something similar have a lot of<br clear="none">
concurrent creation/deletion?<br clear="none">
</blockquote>
<br clear="none">
It was a test of continuously deleting and creating VMs, with this pseudocode:<br clear="none">
<br clear="none">
thread_pool = new_thread_pool(size=30)<br clear="none">
for x in range(0,30):<br clear="none">
thread_pool.submit(create_vm)<br clear="none">
thread_pool.wait_for_all_threads_to_complete()<br clear="none">
while True:<br clear="none">
time.sleep(5)<br clear="none">
for x in range(0,int(random.random()*5)):<br clear="none">
thread_pool.submit(randomly_delete_a_vm_and_create_a_new_one)<br clear="none">
<br clear="none">
I'm not clear whether that would qualify as 'concurrent', in the sense that you have in mind.<br clear="none">
<br clear="none">
Regards,<br clear="none">
Neil<br clear="none">
<br clear="none">
<blockquote class="yiv8713744027gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;"><span class="yiv8713744027">
On Mon, Jun 8, 2015 at 12:21 PM, Andrew Woodward <<a rel="nofollow" shape="rect" ymailto="mailto:awoodward@mirantis.com" target="_blank" href="mailto:awoodward@mirantis.com">awoodward@mirantis.com</a><br clear="none"></span><span class="yiv8713744027">
<mailto:<a rel="nofollow" shape="rect" ymailto="mailto:awoodward@mirantis.com" target="_blank" href="mailto:awoodward@mirantis.com">awoodward@mirantis.com</a>>> wrote:<br clear="none">
<br clear="none">
Daniel,<br clear="none">
<br clear="none">
This sounds familiar, see if this matches [1]. IIRC, there was<br clear="none">
another issue like this that was might already address this in the<br clear="none">
updates into Fuel 5.1.2 packages repo [2]. You can either update the<br clear="none">
neutron packages from [2] Or try one of community builds for 5.1.2<br clear="none">
[3]. If this doesn't resolve the issue, open a bug against MOS dev [4].<br clear="none">
<br clear="none">
[1] <a rel="nofollow" shape="rect" target="_blank" href="https://bugs.launchpad.net/bugs/1295715">https://bugs.launchpad.net/bugs/1295715</a><br clear="none">
[2] <a rel="nofollow" shape="rect" target="_blank" href="http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/">http://fuel-repository.mirantis.com/fwm/5.1.2/ubuntu/pool/main/</a><br clear="none">
[3] <a rel="nofollow" shape="rect" target="_blank" href="https://ci.fuel-infra.org/">https://ci.fuel-infra.org/</a><br clear="none">
[4] <a rel="nofollow" shape="rect" target="_blank" href="https://bugs.launchpad.net/mos/+filebug">https://bugs.launchpad.net/mos/+filebug</a><br clear="none">
<br clear="none">
On Mon, Jun 8, 2015 at 10:15 AM Neil Jerram<br clear="none"></span><div><div class="yiv8713744027h5">
<<a rel="nofollow" shape="rect" ymailto="mailto:Neil.Jerram@metaswitch.com" target="_blank" href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a> <mailto:<a rel="nofollow" shape="rect" ymailto="mailto:Neil.Jerram@metaswitch.com" target="_blank" href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a>>> wrote:<br clear="none">
<br clear="none">
Two further thoughts on this:<br clear="none">
<br clear="none">
1. Another DHCP agent problem that my team noticed is that it<br clear="none">
call_driver('reload_allocations') takes a bit of time (to<br clear="none">
regenerate the<br clear="none">
Dnsmasq config files, and to spawn a shell that sends a HUP<br clear="none">
signal) -<br clear="none">
enough so that if there is a fast steady rate of port-create and<br clear="none">
port-delete notifications coming from the Neutron server, these can<br clear="none">
build up in DHCPAgent's RPC queue, and then they still only get<br clear="none">
dispatched one at a time. So the queue and the time delay<br clear="none">
become longer<br clear="none">
and longer.<br clear="none">
<br clear="none">
I have a fix pending for this, which uses an extra thread to<br clear="none">
read those<br clear="none">
notifications off the RPC queue onto an internal queue, and then<br clear="none">
batches<br clear="none">
the call_driver('reload_allocations') processing when there is a<br clear="none">
contiguous sequence of such notifications - i.e. only does the<br clear="none">
config<br clear="none">
regeneration and HUP once, instead of lots of times.<br clear="none">
<br clear="none">
I don't think this is directly related to what you are seeing - but<br clear="none">
perhaps there actually is some link that I am missing.<br clear="none">
<br clear="none">
2. There is an interesting and vaguely similar thread currently<br clear="none">
being<br clear="none">
discussed about the L3 agent (subject "L3 agent rescheduling<br clear="none">
issue") -<br clear="none">
about possible RPC/threading issues between the agent and the<br clear="none">
Neutron<br clear="none">
server. You might like to review that thread and see if it<br clear="none">
describes<br clear="none">
any problems analogous to your DHCP one.<br clear="none">
<br clear="none">
Regards,<br clear="none">
Neil<br clear="none">
<br clear="none">
<br clear="none">
On 08/06/15 17:53, Neil Jerram wrote:<br clear="none">
> My team has seen a problem that could be related: in a churn<br clear="none">
test where<br clear="none">
> VMs are created and terminated at a constant rate - but so<br clear="none">
that the<br clear="none">
> number of active VMs should remain roughly constant - the<br clear="none">
size of the<br clear="none">
> host and addn_hosts files keeps increasing.<br clear="none">
><br clear="none">
> In other words, it appears that the config for VMs that have<br clear="none">
actually<br clear="none">
> been terminated is not being removed from the config file.<br clear="none">
Clearly, if<br clear="none">
> you have a limited pool of IP addresses, this can eventually<br clear="none">
lead to the<br clear="none">
> problem that you have described.<br clear="none">
><br clear="none">
> For your case - i.e. with Icehouse - the problem might be<br clear="none">
> <a rel="nofollow" shape="rect" target="_blank" href="https://bugs.launchpad.net/neutron/+bug/1192381">https://bugs.launchpad.net/neutron/+bug/1192381</a>. I'm not<br clear="none">
sure if the<br clear="none">
> fix for that problem - i.e. sending port-create and port-delete<br clear="none">
> notifications to DHCP agents even when the server thinks they<br clear="none">
are down -<br clear="none">
> was merged before the Icehouse release, or not.<br clear="none">
><br clear="none">
> But there must be at least one other cause as well, because<br clear="none">
my team was<br clear="none">
> seeing this with Juno-level code.<br clear="none">
><br clear="none">
> Therefore I, too, would be interested in any other insights<br clear="none">
about this<br clear="none">
> problem.<br clear="none">
><br clear="none">
> Regards,<br clear="none">
> Neil<br clear="none">
><br clear="none">
><br clear="none">
><br clear="none">
> On 08/06/15 16:26, Daniel Comnea wrote:<br clear="none">
>> Any help, ideas please?<br clear="none">
>><br clear="none">
>> Thx,<br clear="none">
>> Dani<br clear="none">
>><br clear="none">
>> On Mon, Jun 8, 2015 at 9:25 AM, Daniel Comnea<br clear="none">
<<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a> <mailto:<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a>><br clear="none"></div></div>
>> <mailto:<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a><span class="yiv8713744027"><br clear="none">
<mailto:<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a>>>> wrote:<br clear="none">
>><br clear="none">
>> + Operators<br clear="none">
>><br clear="none">
>> Much thanks in advance,<br clear="none">
>> Dani<br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
>> On Sun, Jun 7, 2015 at 6:31 PM, Daniel Comnea<br clear="none">
<<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a> <mailto:<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a>><br clear="none"></span>
>> <mailto:<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a><div><div class="yiv8713744027h5"><br clear="none">
<mailto:<a rel="nofollow" shape="rect" ymailto="mailto:comnea.dani@gmail.com" target="_blank" href="mailto:comnea.dani@gmail.com">comnea.dani@gmail.com</a>>>> wrote:<br clear="none">
>><br clear="none">
>> Hi all,<br clear="none">
>><br clear="none">
>> I'm running IceHouse (build using Fuel 5.1.1) on<br clear="none">
Ubuntu where<br clear="none">
>> dnsmask version 2.59-4.<br clear="none">
>> I have a very basic network layout where i have a<br clear="none">
private net<br clear="none">
>> which has 2 subnets<br clear="none">
>><br clear="none">
>> 2fb7de9d-d6df-481f-acca-2f7860cffa60 | private-net<br clear="none">
>> |<br clear="none">
>> e79c3477-d3e5-471c-a728-8d881cf31bee<br clear="none">
<a rel="nofollow" shape="rect" target="_blank" onclick="return theMainWindow.showLinkWarning(this)" href="http://192.168.110.0/24">192.168.110.0/24</a> <<a rel="nofollow" shape="rect" target="_blank" onclick="return theMainWindow.showLinkWarning(this)" href="http://192.168.110.0/24">http://192.168.110.0/24</a>><br clear="none">
>> <<a rel="nofollow" shape="rect" target="_blank" onclick="return theMainWindow.showLinkWarning(this)" href="http://192.168.110.0/24">http://192.168.110.0/24</a>> |<br clear="none">
>> |<br clear="none">
>> |<br clear="none">
|<br clear="none">
>> f48c3223-8507-455c-9c13-8b727ea5f441<br clear="none">
<a rel="nofollow" shape="rect" target="_blank" onclick="return theMainWindow.showLinkWarning(this)" href="http://192.168.111.0/24">192.168.111.0/24</a> <<a rel="nofollow" shape="rect" target="_blank" onclick="return theMainWindow.showLinkWarning(this)" href="http://192.168.111.0/24">http://192.168.111.0/24</a>><br clear="none">
>> <<a rel="nofollow" shape="rect" target="_blank" onclick="return theMainWindow.showLinkWarning(this)" href="http://192.168.111.0/24">http://192.168.111.0/24</a>> |<br clear="none">
>><br clear="none">
>> and i'm creating VMs via HEAT.<br clear="none">
>> What is happening is that sometimes i get duplicated<br clear="none">
entries in<br clear="none">
>> [1] and because of that the VM which was spun up<br clear="none">
doesn't get<br clear="none">
>> an ip.<br clear="none">
>> The Dnsmask processes are running okay [2] and i<br clear="none">
can't see<br clear="none">
>> anything special/ wrong in it.<br clear="none">
>><br clear="none">
>> Any idea why this is happening? Or are you aware of<br clear="none">
any bugs<br clear="none">
>> around this area? Do you see a problems with having<br clear="none">
2 subnets<br clear="none">
>> mapped to 1 private-net?<br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
>> Thanks,<br clear="none">
>> Dani<br clear="none">
>><br clear="none">
>> [1]<br clear="none">
>><br clear="none">
>><br clear="none">
/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts<br clear="none">
>><br clear="none">
>> [2]<br clear="none">
>><br clear="none">
>> nobody 5664 1 0 Jun02 ? 00:00:08 dnsmasq<br clear="none">
>> --no-hosts --no-resolv --strict-order --bind-interfaces<br clear="none">
>> --interface=tapc9164734-0c --except-interface=lo<br clear="none">
>><br clear="none">
>><br clear="none">
--pid-file=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/pid<br clear="none">
>><br clear="none">
>><br clear="none">
--dhcp-hostsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/host<br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
--addn-hosts=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/addn_hosts<br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
--dhcp-optsfile=/var/lib/neutron/dhcp/2fb7de9d-d6df-481f-acca-2f7860cffa60/opts<br clear="none">
>><br clear="none">
>> --leasefile-ro --dhcp-authoritative<br clear="none">
>> --dhcp-range=set:tag0,192.168.110.0,static,86400s<br clear="none">
>> --dhcp-range=set:tag1,192.168.111.0,static,86400s<br clear="none">
>> --dhcp-lease-max=512 --conf-file= --server=10.0.0.31<br clear="none">
>> --server=10.0.0.32 --domain=openstacklocal<br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
>><br clear="none">
>> _______________________________________________<br clear="none">
>> OpenStack-operators mailing list<br clear="none">
>> <a rel="nofollow" shape="rect" ymailto="mailto:OpenStack-operators@lists.openstack.org" target="_blank" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br clear="none"></div></div>
<mailto:<a rel="nofollow" shape="rect" ymailto="mailto:OpenStack-operators@lists.openstack.org" target="_blank" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a>><span class="yiv8713744027"><br clear="none">
>><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br clear="none">
>><br clear="none">
><br clear="none">
> _______________________________________________<br clear="none">
> OpenStack-operators mailing list<br clear="none">
> <a rel="nofollow" shape="rect" ymailto="mailto:OpenStack-operators@lists.openstack.org" target="_blank" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br clear="none"></span>
<mailto:<a rel="nofollow" shape="rect" ymailto="mailto:OpenStack-operators@lists.openstack.org" target="_blank" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a>><span class="yiv8713744027"><br clear="none">
><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br clear="none">
<br clear="none">
__________________________________________________________________________<br clear="none">
OpenStack Development Mailing List (not for usage questions)<br clear="none">
Unsubscribe:<br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br clear="none"></span>
<<a rel="nofollow" shape="rect" target="_blank" href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span class="yiv8713744027"><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br clear="none">
<br clear="none">
--<br clear="none">
--<br clear="none">
Andrew Woodward<br clear="none">
Mirantis<br clear="none">
Fuel Community Ambassador<br clear="none">
Ceph Community<br clear="none">
<br clear="none">
__________________________________________________________________________<br clear="none">
OpenStack Development Mailing List (not for usage questions)<br clear="none">
Unsubscribe:<br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br clear="none"></span>
<<a rel="nofollow" shape="rect" target="_blank" href="http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span class="yiv8713744027"><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><div class="yiv8713744027yqt4160899112" id="yiv8713744027yqtfd23292"><br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
<br clear="none">
--<br clear="none">
Kevin Benton<br clear="none">
<br clear="none">
<br clear="none"></div></span><div class="yiv8713744027yqt4160899112" id="yiv8713744027yqtfd18315"><span class="yiv8713744027">
_______________________________________________<br clear="none">
OpenStack-operators mailing list<br clear="none">
<a rel="nofollow" shape="rect" ymailto="mailto:OpenStack-operators@lists.openstack.org" target="_blank" href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a><br clear="none">
<a rel="nofollow" shape="rect" target="_blank" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators</a><br clear="none">
<br clear="none">
</span></div></blockquote><div class="yiv8713744027yqt4160899112" id="yiv8713744027yqtfd44335">
</div></blockquote></div><div class="yiv8713744027yqt4160899112" id="yiv8713744027yqtfd84602"><br clear="none"><br clear="all"><div><br clear="none"></div>-- <br clear="none"><div class="yiv8713744027gmail_signature"><div>Kevin Benton</div></div>
</div></div></div></div><br><div class="yqt4160899112" id="yqtfd74967">__________________________________________________________________________<br clear="none">OpenStack Development Mailing List (not for usage questions)<br clear="none">Unsubscribe: <a shape="rect" ymailto="mailto:OpenStack-dev-request@lists.openstack.org" href="mailto:OpenStack-dev-request@lists.openstack.org">OpenStack-dev-request@lists.openstack.org</a>?subject:unsubscribe<br clear="none"><a shape="rect" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br clear="none"></div><br><br></div> </div> </div> </div></div></body></html>