[openstack-dev] [neutron]Performance of security group
shihanzhang
ayshihanzhang at 126.com
Thu Jul 10 12:15:30 UTC 2014
Yes, but I have modified that patch, I will commit it as soon as possilbe!
Best regards,
shihanzhang
At 2014-07-10 07:06:39, "Miguel Angel Ajo Pelayo" <mangelajo at redhat.com> wrote:
>Wow Shihanzhang, truly awesome!,
>
> Is the current implementation for https://review.openstack.org/#/c/104522/
>also ready?, could you make it available?.
>
>
>Good work!,
>
>
>----- Original Message -----
>>
>> With the deployment 'nova + neutron + openvswitch', when we bulk create about
>> 500 VM with a default security group, the CPU usage of neutron-server and
>> openvswitch agent is very high, especially the CPU usage of openvswitch
>> agent will be 100%, this will cause creating VMs failed.
>>
>> With the method discussed in mailist:
>> 1) ipset optimization (https://review.openstack.org/#/c/100761/)
>> 3) sg rpc optimization (with fanout) (
>> https://review.openstack.org/#/c/104522/)
>> I have implement these two scheme in my deployment, when we again bulk
>> create about 500 VM with a default security group, the CPU usage of
>> openvswitch agent will reduce to 10%, e ven lower than 10%, so I think the i
>> provement of these two options are very efficient.
>> Who can help us to review our spec?
>> Best regards,
>> shihanzhang
>>
>>
>>
>>
>>
>> At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
>> >-----BEGIN PGP SIGNED MESSAGE-----
>> >Hash: SHA512
>> >
>> >Oh, so you have the enhancement implemented? Great! Any numbers that
>> >shows how much we gain from that?
>> >
>> >/Ihar
>> >
>> >On 03/07/14 02:49, shihanzhang wrote:
>> >> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>> >> I will modify my spec, when the spec is approved, I will commit the
>> >> codes as soon as possilbe!
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
>> >> wrote:
>> >>>
>> >>> Nice Shihanzhang,
>> >>>
>> >>> Do you mean the ipset implementation is ready, or just the
>> >>> spec?.
>> >>>
>> >>>
>> >>> For the SG group refactor, I don't worry about who does it, or
>> >>> who takes the credit, but I believe it's important we address
>> >>> this bottleneck during Juno trying to match nova's scalability.
>> >>>
>> >>> Best regards, Miguel Ángel.
>> >>>
>> >>>
>> >>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>> >>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>> >>>> split the work in several specs, I have finished the work (
>> >>>> ipset optimization), you can do 'sg rpc optimization (without
>> >>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>> >>>> I think we need talk about it, because just using ipset to
>> >>>> optimize security group agent codes does not bring the best
>> >>>> results!
>> >>>>
>> >>>> Best regards, shihanzhang.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
>> >>>> wrote:
>> >> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>> >>
>> >>> Shihazhang,
>> >>
>> >>> I really believe we need the RPC refactor done for this cycle,
>> >>> and given the close deadlines we have (July 10 for spec
>> >>> submission and July 20 for spec approval).
>> >>
>> >>> Don't you think it's going to be better to split the work in
>> >>> several specs?
>> >>
>> >>> 1) ipset optimization (you) 2) sg rpc optimization (without
>> >>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>> >>> , me)
>> >>
>> >>
>> >>> This way we increase the chances of having part of this for the
>> >>> Juno cycle. If we go for something too complicated is going to
>> >>> take more time for approval.
>> >>
>> >>
>> >> I agree. And it not only increases chances to get at least some of
>> >> those highly demanded performance enhancements to get into Juno,
>> >> it's also "the right thing to do" (c). It's counterproductive to
>> >> put multiple vaguely related enhancements in single spec. This
>> >> would dim review focus and put us into position of getting
>> >> 'all-or-nothing'. We can't afford that.
>> >>
>> >> Let's leave one spec per enhancement. @Shihazhang, what do you
>> >> think?
>> >>
>> >>
>> >>> Also, I proposed the details of "2", trying to bring awareness
>> >>> on the topic, as I have been working with the scale lab in Red
>> >>> Hat to find and understand those issues, I have a very good
>> >>> knowledge of the problem and I believe I could make a very fast
>> >>> advance on the issue at the RPC level.
>> >>
>> >>> Given that, I'd like to work on this specific part, whether or
>> >>> not we split the specs, as it's something we believe critical
>> >>> for neutron scalability and thus, *nova parity*.
>> >>
>> >>> I will start a separate spec for "2", later on, if you find it
>> >>> ok, we keep them as separate ones, if you believe having just 1
>> >>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>> >>> can incorporate my spec in yours, but then
>> >>> "add-ipset-to-security" is not a good spec title to put all this
>> >>> together.
>> >>
>> >>
>> >>> Best regards, Miguel Ángel.
>> >>
>> >>
>> >>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>> >>>>
>> >>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>> >>>> spes, but I will also optimization the RPC from security group
>> >>>> agent to neutron server. Now the modle is
>> >>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>> >>>> sg2..]', this can reduce the size of RPC respose message from
>> >>>> neutron server to security group agent.
>> >>>>
>> >>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>> >>>> <mangelajo at redhat.com> wrote:
>> >>>>>
>> >>>>>
>> >>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>> >>>>> work into this problem, I could file an specific spec for
>> >>>>> the security group RPC optimization, a masterplan in two
>> >>>>> steps:
>> >>>>>
>> >>>>> 1) Refactor the current RPC communication for
>> >>>>> security_groups_for_devices, which could be used for full
>> >>>>> syncs, etc..
>> >>>>>
>> >>>>> 2) Benchmark && make use of a fanout queue per security
>> >>>>> group to make sure only the hosts with instances on a
>> >>>>> certain security group get the updates as they happen.
>> >>>>>
>> >>>>> @shihanzhang do you find it reasonable?
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> ----- Original Message -----
>> >>>>>> ----- Original Message -----
>> >>>>>>> @Nachi: Yes that could a good improvement to factorize
>> >>>>>>> the RPC
>> >>>>>> mechanism.
>> >>>>>>>
>> >>>>>>> Another idea: What about creating a RPC topic per
>> >>>>>>> security group (quid of the
>> >>>>>> RPC topic
>> >>>>>>> scalability) on which an agent subscribes if one of its
>> >>>>>>> ports is
>> >>>>>> associated
>> >>>>>>> to the security group?
>> >>>>>>>
>> >>>>>>> Regards, Édouard.
>> >>>>>>>
>> >>>>>>>
>> >>
>> >>
>> >>>>>> Hmm, Interesting,
>> >>
>> >>>>>> @Nachi, I'm not sure I fully understood:
>> >>
>> >>
>> >>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>> >>>>>> port[SG_ID1, SG_ID2], port2 , port3
>> >>
>> >>
>> >>>>>> Probably we may need to include also the SG_IP_LIST =
>> >>>>>> [SG_IP1, SG_IP2] ...
>> >>
>> >>
>> >>>>>> and let the agent do all the combination work.
>> >>
>> >>>>>> Something like this could make sense?
>> >>
>> >>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>> >>>>>> SG2:{IPs:[....],RULES:[....]} }
>> >>
>> >>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>> >>
>> >>
>> >>>>>> @Edouard, actually I like the idea of having the agent
>> >>>>>> subscribed to security groups they have ports on... That
>> >>>>>> would remove the need to include all the security groups
>> >>>>>> information on every call...
>> >>
>> >>>>>> But would need another call to get the full information of
>> >>>>>> a set of security groups at start/resync if we don't
>> >>>>>> already have any.
>> >>
>> >>
>> >>>>>>>
>> >>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>> >>>>>> ayshihanzhang at 126.com >
>> >>>>>>> wrote:
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> hi Miguel Ángel, I am very agree with you about the
>> >>>>>>> following point:
>> >>>>>>>> * physical implementation on the hosts (ipsets,
>> >>>>>>>> nftables, ... )
>> >>>>>>> --this can reduce the load of compute node.
>> >>>>>>>> * rpc communication mechanisms.
>> >>>>>>> -- this can reduce the load of neutron server can you
>> >>>>>>> help me to review my BP specs?
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>> >>>>>> mangelajo at redhat.com >
>> >>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>> Hi it's a very interesting topic, I was getting ready
>> >>>>>>>> to raise the same concerns about our security groups
>> >>>>>>>> implementation,
>> >>>>>> shihanzhang
>> >>>>>>>> thank you for starting this topic.
>> >>>>>>>>
>> >>>>>>>> Not only at low level where (with our default security
>> >>>>>>>> group rules -allow all incoming from 'default' sg- the
>> >>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>> >>>>>>>> "security_group_rules_for_devices" rpc call from
>> >>>>>>>> ovs-agent to neutron-server grows to message sizes of
>> >>>>>>>>> 100MB,
>> >>>>>>>> generating serious scalability issues or
>> >>>>>>>> timeouts/retries that totally break neutron service.
>> >>>>>>>>
>> >>>>>>>> (example trace of that RPC call with a few instances
>> >>>>>>>> http://www.fpaste.org/104401/14008522/ )
>> >>>>>>>>
>> >>>>>>>> I believe that we also need to review the RPC calling
>> >>>>>>>> mechanism for the OVS agent here, there are several
>> >>>>>>>> possible approaches to
>> >>>>>> breaking
>> >>>>>>>> down (or/and CIDR compressing) the information we
>> >>>>>>>> return via this
>> >>>>>> api
>> >>>>>>>> call.
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> So we have to look at two things here:
>> >>>>>>>>
>> >>>>>>>> * physical implementation on the hosts (ipsets,
>> >>>>>>>> nftables, ... ) * rpc communication mechanisms.
>> >>>>>>>>
>> >>>>>>>> Best regards, Miguel Ángel.
>> >>>>>>>>
>> >>>>>>>> ----- Mensaje original -----
>> >>>>>>>>
>> >>>>>>>>> Do you though about nftables that will replace
>> >>>>>> {ip,ip6,arp,eb}tables?
>> >>>>>>>>> It also based on the rule set mechanism. The issue
>> >>>>>>>>> in that proposition, it's only stable since the
>> >>>>>>>>> begin
>> >>>>>> of the
>> >>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>> >>>>>>>>> I don't list here (leverage iptables
>> >>>>>> limitation,
>> >>>>>>>>> efficient update rule, rule set, standardization of
>> >>>>>>>>> netfilter commands...).
>> >>>>>>>>
>> >>>>>>>>> Édouard.
>> >>>>>>>>
>> >>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>> >>>>>>>>> henry4hly at gmail.com > wrote:
>> >>>>>>>>
>> >>>>>>>>>> we have done some tests, but have different
>> >>>>>>>>>> result: the
>> >>>>>> performance is
>> >>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>> >>>>>>>>>> but huge gap between enable/disable iptable hook
>> >>>>>>>>>> on linux bridge
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>> >>>>>> ayshihanzhang at 126.com
>> >>>>>>>>>>>
>> >>>>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>> Now I have not get accurate test data, but I can
>> >>>>>>>>>>> confirm the following points?
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>> >>>>>>>>>>> is liner,
>> >>>>>> iptable
>> >>>>>>>>>>> filter it one by one, if a VM in default
>> >>>>>>>>>>> security group and this default security group
>> >>>>>>>>>>> have many members, but ipset chain is set, the
>> >>>>>>>>>>> time
>> >>>>>> ipset
>> >>>>>>>>>>> filter one and many member is not much
>> >>>>>>>>>>> difference.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>> 2. when the iptable rule is very large, the
>> >>>>>>>>>>> probability of
>> >>>>>> failure
>> >>>>>>>>>>> that iptable-save save the iptable rule is very
>> >>>>>>>>>>> large.
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>> >>>>>>>>>>> blak111 at gmail.com
>> >>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>>> This sounds like a good idea to handle some of
>> >>>>>>>>>>>> the
>> >>>>>> performance
>> >>>>>>>>>>>> issues until the ovs firewall can be
>> >>>>>>>>>>>> implemented down the the line.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>> Do you have any performance comparisons?
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>> >>>>>> ayshihanzhang at 126.com >
>> >>>>>>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>>>> Hello all,
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>>>> Now in neutron, it use iptable implementing
>> >>>>>>>>>>>>> security
>> >>>>>> group, but
>> >>>>>>>>>>>>> the performance of this implementation is
>> >>>>>>>>>>>>> very poor, there
>> >>>>>> is a bug:
>> >>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >to
>> >>>>>> reflect this
>> >>>>>>>>>>>>> problem. In his test, w ith default security
>> >>>>>>>>>>>>> groups(which has remote security group),
>> >>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>> >>>>>>>>>>>>> Iptable rules
>> >>>>>> on evry
>> >>>>>>>>>>>>> compute node, although his patch can reduce
>> >>>>>>>>>>>>> the processing time, but
>> >>>>>> it don't
>> >>>>>>>>>>>>> solve this problem fundamentally. I have
>> >>>>>>>>>>>>> commit a BP to solve this
>> >>>>>> problem:
>> >>>>>>>>>>>>>
>> >>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>> >>
>> >>>>>>
>> >>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>> There are other people interested in this
>> >>>>>>>>>>>>> it?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>>>> _______________________________________________
>> >>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>>
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>>>>>
>> >>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>> >>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>
>> >>>>>>
>> >>>
>> >>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>
>> >>>>>>>>>> _______________________________________________
>> >>>>>>>>>
>> >>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>
>> >>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>> >>>>>>>>>>
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>
>> >>>>>>
>> >>>
>> >>>>>>>>
>> >>>>>>>>> _______________________________________________
>> >>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>> OpenStack-dev at lists.openstack.org >>
>> >>>>>>>>>
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>
>> >>>>>>
>> >>
>> >>>>>>>> _______________________________________________
>> >>>>>>>> OpenStack-dev mailing list
>> >>>>>>>> OpenStack-dev at lists.openstack.org >
>> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>>>>>>>
>> >>>>>>>>
>> >>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> OpenStack-dev mailing list
>> >>>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>>>>>>
>> >>>>>>>
>> >>
>> >>>>>>>
>> >>>>>>>
>> >>>>>>> _______________________________________________
>> >>>>>>> OpenStack-dev mailing list
>> >>>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>>>>>>
>> >>>>>>>
>> >>
>> >>
>> >>>>>> _______________________________________________
>> >>>>>> OpenStack-dev mailing list
>> >>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>>>>>
>> >>>>>
>> >>>>>
>> >>
>> >> _______________________________________________
>> >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>>
>> >>
>> >>>>>
>> >_______________________________________________
>> >>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>
>> >>
>> >>>>
>> >>>>
>> >>> _______________________________________________ OpenStack-dev
>> >>> mailing list OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>
>> >>>>>_______________________________________________
>> >>>>>OpenStack-dev
>> >>>
>> >mailing list
>> >>>>> OpenStack-dev at lists.openstack.org
>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>>
>> >_______________________________________________
>> >>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>
>> >>>
>> >>>_______________________________________________
>> >>>OpenStack-dev
>> >>>>
>> >mailing list
>> >>> OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>>
>> >>
>> >>
>> >>
>> >>
>> >> _______________________________________________ OpenStack-dev
>> >> mailing list OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >-----BEGIN PGP SIGNATURE-----
>> >Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>> >Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>> >
>> >iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
>> >IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
>> >WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
>> >7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
>> >1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
>> >/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
>> >=TxAT
>> >-----END PGP SIGNATURE-----
>> >
>> >_______________________________________________
>> >OpenStack-dev mailing list
>> >OpenStack-dev at lists.openstack.org
>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/a85ddd99/attachment.html>
More information about the OpenStack-dev
mailing list