Open Stack

On 08/20/2014 07:34 AM, Miguel Angel Ajo Pelayo wrote:
> I couldn't resist making a little benchmark test of the new RPC implementation
> shihanzhang wrote:
>
> http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite
>
> The results are awesome :-)

Indeed, fantastic news. ++

-jay

> We yet need to polish the tests a bit, and it's ready.
>
> Best regards,
> Miguel Ángel.
>
> ----- Original Message -----
>> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzhang at 126.com> wrote:
>>>
>>> With the deployment 'nova + neutron + openvswitch', when we bulk create
>>> about 500 VM with a default security group, the CPU usage of neutron-server
>>> and openvswitch agent is very high, especially the CPU usage of openvswitch
>>> agent will be 100%, this will cause creating VMs failed.
>>>
>>> With the method discussed in mailist:
>>>
>>> 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
>>>
>>> 3) sg rpc optimization (with fanout)
>>> (https://review.openstack.org/#/c/104522/)
>>>
>>> I have implement  these two scheme in my deployment,  when we again bulk
>>> create about 500 VM with a default security group, the CPU usage of
>>> openvswitch agent will reduce to 10%, even lower than 10%, so I think the
>>> iprovement of these two options are very efficient.
>>>
>>> Who can help us to review our spec?
>>>
>> This is great work! These are on my list of things to review in detail
>> soon, but given the Neutron sprint this week, I haven't had time yet.
>> I'll try to remedy that by the weekend.
>>
>> Thanks!
>> Kyle
>>
>>>     Best regards,
>>> shihanzhang
>>>
>>>
>>>
>>>
>>>
>>> At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
>>>> -----BEGIN PGP SIGNED MESSAGE-----
>>>> Hash: SHA512
>>>>
>>>> Oh, so you have the enhancement implemented? Great! Any numbers that
>>>> shows how much we gain from that?
>>>>
>>>> /Ihar
>>>>
>>>> On 03/07/14 02:49, shihanzhang wrote:
>>>>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>>>>> I will modify my spec, when the spec is approved, I will commit the
>>>>> codes as soon as possilbe!
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
>>>>> wrote:
>>>>>>
>>>>>> Nice Shihanzhang,
>>>>>>
>>>>>> Do you mean the ipset implementation is ready, or just the
>>>>>> spec?.
>>>>>>
>>>>>>
>>>>>> For the SG group refactor, I don't worry about who does it, or
>>>>>> who takes the credit, but I believe it's important we address
>>>>>> this bottleneck during Juno trying to match nova's scalability.
>>>>>>
>>>>>> Best regards, Miguel Ángel.
>>>>>>
>>>>>>
>>>>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>>>>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>>>>>>> split  the work in several specs, I have finished the work (
>>>>>>> ipset optimization), you can do 'sg rpc optimization (without
>>>>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>>>>>>> I think we need talk about it, because just using ipset to
>>>>>>> optimize security group agent codes does not bring the best
>>>>>>> results!
>>>>>>>
>>>>>>> Best regards, shihanzhang.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
>>>>>>> wrote:
>>>>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>>>>
>>>>>> Shihazhang,
>>>>>
>>>>>> I really believe we need the RPC refactor done for this cycle,
>>>>>> and given the close deadlines we have (July 10 for spec
>>>>>> submission and July 20 for spec approval).
>>>>>
>>>>>> Don't you think it's going to be better to split the work in
>>>>>> several specs?
>>>>>
>>>>>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>>>>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>>>>>> , me)
>>>>>
>>>>>
>>>>>> This way we increase the chances of having part of this for the
>>>>>> Juno cycle. If we go for something too complicated is going to
>>>>>> take more time for approval.
>>>>>
>>>>>
>>>>> I agree. And it not only increases chances to get at least some of
>>>>> those highly demanded performance enhancements to get into Juno,
>>>>> it's also "the right thing to do" (c). It's counterproductive to
>>>>> put multiple vaguely related enhancements in single spec. This
>>>>> would dim review focus and put us into position of getting
>>>>> 'all-or-nothing'. We can't afford that.
>>>>>
>>>>> Let's leave one spec per enhancement. @Shihazhang, what do you
>>>>> think?
>>>>>
>>>>>
>>>>>> Also, I proposed the details of "2", trying to bring awareness
>>>>>> on the topic, as I have been working with the scale lab in Red
>>>>>> Hat to find and understand those issues, I have a very good
>>>>>> knowledge of the problem and I believe I could make a very fast
>>>>>> advance on the issue at the RPC level.
>>>>>
>>>>>> Given that, I'd like to work on this specific part, whether or
>>>>>> not we split the specs, as it's something we believe critical
>>>>>> for neutron scalability and thus, *nova parity*.
>>>>>
>>>>>> I will start a separate spec for "2", later on, if you find it
>>>>>> ok, we keep them as separate ones, if you believe having just 1
>>>>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>>>>>> can incorporate my spec in yours, but then
>>>>>> "add-ipset-to-security" is not a good spec title to put all this
>>>>>> together.
>>>>>
>>>>>
>>>>>> Best regards, Miguel Ángel.
>>>>>
>>>>>
>>>>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>>>>>
>>>>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>>>>>>> spes, but I will also optimization the RPC from security group
>>>>>>> agent to neutron server. Now the modle is
>>>>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>>>>>>> sg2..]', this can reduce the size of RPC respose message from
>>>>>>> neutron server to security group agent.
>>>>>>>
>>>>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>>>>>>> <mangelajo at redhat.com> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>>>>>>>> work into this problem, I could file an specific spec for
>>>>>>>> the security group RPC optimization, a masterplan in two
>>>>>>>> steps:
>>>>>>>>
>>>>>>>> 1) Refactor the current RPC communication for
>>>>>>>> security_groups_for_devices, which could be used for full
>>>>>>>> syncs, etc..
>>>>>>>>
>>>>>>>> 2) Benchmark && make use of a fanout queue per security
>>>>>>>> group to make sure only the hosts with instances on a
>>>>>>>> certain security group get the updates as they happen.
>>>>>>>>
>>>>>>>> @shihanzhang do you find it reasonable?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> ----- Original Message -----
>>>>>>>>>> @Nachi: Yes that could a good improvement to factorize
>>>>>>>>>> the RPC
>>>>>>>>> mechanism.
>>>>>>>>>>
>>>>>>>>>> Another idea: What about creating a RPC topic per
>>>>>>>>>> security group (quid of the
>>>>>>>>> RPC topic
>>>>>>>>>> scalability) on which an agent subscribes if one of its
>>>>>>>>>> ports is
>>>>>>>>> associated
>>>>>>>>>> to the security group?
>>>>>>>>>>
>>>>>>>>>> Regards, Édouard.
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>>>>
>>>>>>>>> Hmm, Interesting,
>>>>>
>>>>>>>>> @Nachi, I'm not sure I fully understood:
>>>>>
>>>>>
>>>>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>>>>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>>>>>
>>>>>
>>>>>>>>> Probably we may need to include also the SG_IP_LIST =
>>>>>>>>> [SG_IP1, SG_IP2] ...
>>>>>
>>>>>
>>>>>>>>> and let the agent do all the combination work.
>>>>>
>>>>>>>>> Something like this could make sense?
>>>>>
>>>>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>>>>>>>>> SG2:{IPs:[....],RULES:[....]} }
>>>>>
>>>>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>>>>>
>>>>>
>>>>>>>>> @Edouard, actually I like the idea of having the agent
>>>>>>>>> subscribed to security groups they have ports on... That
>>>>>>>>> would remove the need to include all the security groups
>>>>>>>>> information on every call...
>>>>>
>>>>>>>>> But would need another call to get the full information of
>>>>>>>>> a set of security groups at start/resync if we don't
>>>>>>>>> already have any.
>>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>>>>>>>>> ayshihanzhang at 126.com >
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> hi Miguel Ángel, I am very agree with you about the
>>>>>>>>>> following point:
>>>>>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>>>>>> nftables, ... )
>>>>>>>>>> --this can reduce the load of compute node.
>>>>>>>>>>> * rpc communication mechanisms.
>>>>>>>>>> -- this can reduce the load of neutron server can you
>>>>>>>>>> help me to review my BP specs?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>>>>>>>>> mangelajo at redhat.com >
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi it's a very interesting topic, I was getting ready
>>>>>>>>>>> to raise the same concerns about our security groups
>>>>>>>>>>> implementation,
>>>>>>>>> shihanzhang
>>>>>>>>>>> thank you for starting this topic.
>>>>>>>>>>>
>>>>>>>>>>> Not only at low level where (with our default security
>>>>>>>>>>> group rules -allow all incoming from 'default' sg- the
>>>>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>>>>>>>>>>> "security_group_rules_for_devices" rpc call from
>>>>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>>>>>>>>>>>> 100MB,
>>>>>>>>>>> generating serious scalability issues or
>>>>>>>>>>> timeouts/retries that totally break neutron service.
>>>>>>>>>>>
>>>>>>>>>>> (example trace of that RPC call with a few instances
>>>>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>>>>>>>>>>>
>>>>>>>>>>> I believe that we also need to review the RPC calling
>>>>>>>>>>> mechanism for the OVS agent here, there are several
>>>>>>>>>>> possible approaches to
>>>>>>>>> breaking
>>>>>>>>>>> down (or/and CIDR compressing) the information we
>>>>>>>>>>> return via this
>>>>>>>>> api
>>>>>>>>>>> call.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> So we have to look at two things here:
>>>>>>>>>>>
>>>>>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>>>>>> nftables, ... ) * rpc communication mechanisms.
>>>>>>>>>>>
>>>>>>>>>>> Best regards, Miguel Ángel.
>>>>>>>>>>>
>>>>>>>>>>> ----- Mensaje original -----
>>>>>>>>>>>
>>>>>>>>>>>> Do you though about nftables that will replace
>>>>>>>>> {ip,ip6,arp,eb}tables?
>>>>>>>>>>>> It also based on the rule set mechanism. The issue
>>>>>>>>>>>> in that proposition, it's only stable since the
>>>>>>>>>>>> begin
>>>>>>>>> of the
>>>>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>>>>>>>>>>>> I don't list here (leverage iptables
>>>>>>>>> limitation,
>>>>>>>>>>>> efficient update rule, rule set, standardization of
>>>>>>>>>>>> netfilter commands...).
>>>>>>>>>>>
>>>>>>>>>>>> Édouard.
>>>>>>>>>>>
>>>>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>>>>>>>>>>>> henry4hly at gmail.com > wrote:
>>>>>>>>>>>
>>>>>>>>>>>>> we have done some tests, but have different
>>>>>>>>>>>>> result: the
>>>>>>>>> performance is
>>>>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>>>>>>>>>>>>> but huge gap between enable/disable iptable hook
>>>>>>>>>>>>> on linux bridge
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>>>>>>>>> ayshihanzhang at 126.com
>>>>>>>>>>>>>>
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> Now I have not get accurate test data, but I can
>>>>>>>>>>>>>> confirm the following points?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>>>>>>>>>>>>>> is liner,
>>>>>>>>> iptable
>>>>>>>>>>>>>> filter it one by one, if a VM in default
>>>>>>>>>>>>>> security group and this default security group
>>>>>>>>>>>>>> have many members, but ipset chain is set, the
>>>>>>>>>>>>>> time
>>>>>>>>> ipset
>>>>>>>>>>>>>> filter one and many member is not much
>>>>>>>>>>>>>> difference.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> 2. when the iptable rule is very large, the
>>>>>>>>>>>>>> probability of
>>>>>>>>> failure
>>>>>>>>>>>>>> that iptable-save save the iptable rule is very
>>>>>>>>>>>>>> large.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>>>>>>>>>>>>>> blak111 at gmail.com
>>>>>>>>>> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>>> This sounds like a good idea to handle some of
>>>>>>>>>>>>>>> the
>>>>>>>>> performance
>>>>>>>>>>>>>>> issues until the ovs firewall can be
>>>>>>>>>>>>>>> implemented down the the line.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>> Do you have any performance comparisons?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>>>>>>>>> ayshihanzhang at 126.com >
>>>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>>>> Now in neutron, it use iptable implementing
>>>>>>>>>>>>>>>> security
>>>>>>>>> group, but
>>>>>>>>>>>>>>>> the performance of this implementation is
>>>>>>>>>>>>>>>> very poor, there
>>>>>>>>> is a bug:
>>>>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>> to
>>>>>>>>> reflect this
>>>>>>>>>>>>>>>> problem. In his test, w ith default security
>>>>>>>>>>>>>>>> groups(which has remote security group),
>>>>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>>>>>>>>>>>>>>>> Iptable rules
>>>>>>>>> on evry
>>>>>>>>>>>>>>>> compute node, although his patch can reduce
>>>>>>>>>>>>>>>> the processing time, but
>>>>>>>>> it don't
>>>>>>>>>>>>>>>> solve this problem fundamentally. I have
>>>>>>>>>>>>>>>> commit a BP to solve this
>>>>>>>>> problem:
>>>>>>>>>>>>>>>>
>>>>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>>>>>
>>>>>>>>>
>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>> There are other people interested in this
>>>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>>>>
>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>
>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>
>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>>>>
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>>>
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>
>>>>>>>>>
>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>> OpenStack-dev at lists.openstack.org >
>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>
>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>
>>>>>>>>
>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>
>>>>>>>
>>>>>>>
>>>>>> _______________________________________________ OpenStack-dev
>>>>>> mailing list OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev
>>>>>>
>>>> mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev
>>>>>>>
>>>> mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________ OpenStack-dev
>>>>> mailing list OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>> -----BEGIN PGP SIGNATURE-----
>>>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>>>
>>>> iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
>>>> IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
>>>> WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
>>>> 7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
>>>> 1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
>>>> /qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
>>>> =TxAT
>>>> -----END PGP SIGNATURE-----
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>