[openstack-dev] [neutron]Performance of security group

shihanzhang ayshihanzhang at 126.com
Thu Aug 21 03:11:18 UTC 2014 

hi neutroner!
    my patch about BP:https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security need install ipset in devstack, I have commit the patch:https://review.openstack.org/#/c/113453/, who can help me review it, thanks very much!


     Best regards,
shihanzhang





At 2014-08-21 10:47:59, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com> wrote:

+1 "NFTablesDriver"!


Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example: https://home.regit.org/2014/02/suricata-and-nftables/


Then, I'm wondering here... What benefits might come for OpenStack Nova / Neutron, if it comes with a NFTables driver, instead of the current IPTables?!


* Efficient Security Group design?
* Better FWaaS, maybe with NAT(44/66) support?
* Native support for IPv6, with the defamed NAT66 built-in, simpler "Floating IP" implementation, for both v4 and v6 networks under a single implementation (I don't like NAT66, I prefer a `routed Floating IPv6` version)?
* Metadata over IPv6 still using NAT(66) (I don't like NAT66), single implementation?
* Suricata-as-a-Service?!


It sounds pretty cool!   :-)



On 20 August 2014 23:16, Baohua Yang <yangbaohua at gmail.com> wrote:

Great！
We met similar problems.
The current mechanisms produce too many iptables rules, and it's hard to debug.
Really look forward to seeing a more efficient security group design.



On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <mestery at noironetworks.com> wrote:

On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzhang at 126.com> wrote:
>
> With the deployment 'nova + neutron + openvswitch', when we bulk create
> about 500 VM with a default security group, the CPU usage of neutron-server
> and openvswitch agent is very high, especially the CPU usage of openvswitch
> agent will be 100%, this will cause creating VMs failed.
>
> With the method discussed in mailist:
>
> 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
>
> 3) sg rpc optimization (with fanout)
> (https://review.openstack.org/#/c/104522/)
>
> I have implement  these two scheme in my deployment,  when we again bulk
> create about 500 VM with a default security group, the CPU usage of
> openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> iprovement of these two options are very efficient.
>
> Who can help us to review our spec?
>

This is great work! These are on my list of things to review in detail
soon, but given the Neutron sprint this week, I haven't had time yet.
I'll try to remedy that by the weekend.

Thanks!
Kyle


>    Best regards,
> shihanzhang
>
>
>
>
>
> At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
>>-----BEGIN PGP SIGNED MESSAGE-----
>>Hash: SHA512
>>
>>Oh, so you have the enhancement implemented? Great! Any numbers that
>>shows how much we gain from that?
>>
>>/Ihar
>>
>>On 03/07/14 02:49, shihanzhang wrote:
>>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>>> I will modify my spec, when the spec is approved, I will commit the
>>> codes as soon as possilbe!
>>>
>>>
>>>
>>>
>>>
>>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
>>> wrote:
>>>>
>>>> Nice Shihanzhang,
>>>>
>>>> Do you mean the ipset implementation is ready, or just the
>>>> spec?.
>>>>
>>>>
>>>> For the SG group refactor, I don't worry about who does it, or
>>>> who takes the credit, but I believe it's important we address
>>>> this bottleneck during Juno trying to match nova's scalability.
>>>>
>>>> Best regards, Miguel Ángel.
>>>>
>>>>
>>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>>>>> split  the work in several specs, I have finished the work (
>>>>> ipset optimization), you can do 'sg rpc optimization (without
>>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>>>>> I think we need talk about it, because just using ipset to
>>>>> optimize security group agent codes does not bring the best
>>>>> results!
>>>>>
>>>>> Best regards, shihanzhang.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
>>>>> wrote:
>>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>>
>>>> Shihazhang,
>>>
>>>> I really believe we need the RPC refactor done for this cycle,
>>>> and given the close deadlines we have (July 10 for spec
>>>> submission and July 20 for spec approval).
>>>
>>>> Don't you think it's going to be better to split the work in
>>>> several specs?
>>>
>>>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>>>> , me)
>>>
>>>
>>>> This way we increase the chances of having part of this for the
>>>> Juno cycle. If we go for something too complicated is going to
>>>> take more time for approval.
>>>
>>>
>>> I agree. And it not only increases chances to get at least some of
>>> those highly demanded performance enhancements to get into Juno,
>>> it's also "the right thing to do" (c). It's counterproductive to
>>> put multiple vaguely related enhancements in single spec. This
>>> would dim review focus and put us into position of getting
>>> 'all-or-nothing'. We can't afford that.
>>>
>>> Let's leave one spec per enhancement. @Shihazhang, what do you
>>> think?
>>>
>>>
>>>> Also, I proposed the details of "2", trying to bring awareness
>>>> on the topic, as I have been working with the scale lab in Red
>>>> Hat to find and understand those issues, I have a very good
>>>> knowledge of the problem and I believe I could make a very fast
>>>> advance on the issue at the RPC level.
>>>
>>>> Given that, I'd like to work on this specific part, whether or
>>>> not we split the specs, as it's something we believe critical
>>>> for neutron scalability and thus, *nova parity*.
>>>
>>>> I will start a separate spec for "2", later on, if you find it
>>>> ok, we keep them as separate ones, if you believe having just 1
>>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>>>> can incorporate my spec in yours, but then
>>>> "add-ipset-to-security" is not a good spec title to put all this
>>>> together.
>>>
>>>
>>>> Best regards, Miguel Ángel.
>>>
>>>
>>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>>>
>>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>>>>> spes, but I will also optimization the RPC from security group
>>>>> agent to neutron server. Now the modle is
>>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>>>>> sg2..]', this can reduce the size of RPC respose message from
>>>>> neutron server to security group agent.
>>>>>
>>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>>>>> <mangelajo at redhat.com> wrote:
>>>>>>
>>>>>>
>>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>>>>>> work into this problem, I could file an specific spec for
>>>>>> the security group RPC optimization, a masterplan in two
>>>>>> steps:
>>>>>>
>>>>>> 1) Refactor the current RPC communication for
>>>>>> security_groups_for_devices, which could be used for full
>>>>>> syncs, etc..
>>>>>>
>>>>>> 2) Benchmark && make use of a fanout queue per security
>>>>>> group to make sure only the hosts with instances on a
>>>>>> certain security group get the updates as they happen.
>>>>>>
>>>>>> @shihanzhang do you find it reasonable?
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> ----- Original Message -----
>>>>>>>> @Nachi: Yes that could a good improvement to factorize
>>>>>>>> the RPC
>>>>>>> mechanism.
>>>>>>>>
>>>>>>>> Another idea: What about creating a RPC topic per
>>>>>>>> security group (quid of the
>>>>>>> RPC topic
>>>>>>>> scalability) on which an agent subscribes if one of its
>>>>>>>> ports is
>>>>>>> associated
>>>>>>>> to the security group?
>>>>>>>>
>>>>>>>> Regards, Édouard.
>>>>>>>>
>>>>>>>>
>>>
>>>
>>>>>>> Hmm, Interesting,
>>>
>>>>>>> @Nachi, I'm not sure I fully understood:
>>>
>>>
>>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>>>
>>>
>>>>>>> Probably we may need to include also the SG_IP_LIST =
>>>>>>> [SG_IP1, SG_IP2] ...
>>>
>>>
>>>>>>> and let the agent do all the combination work.
>>>
>>>>>>> Something like this could make sense?
>>>
>>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>>>>>>> SG2:{IPs:[....],RULES:[....]} }
>>>
>>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>>>
>>>
>>>>>>> @Edouard, actually I like the idea of having the agent
>>>>>>> subscribed to security groups they have ports on... That
>>>>>>> would remove the need to include all the security groups
>>>>>>> information on every call...
>>>
>>>>>>> But would need another call to get the full information of
>>>>>>> a set of security groups at start/resync if we don't
>>>>>>> already have any.
>>>
>>>
>>>>>>>>
>>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>>>>>>> ayshihanzhang at 126.com >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> hi Miguel Ángel, I am very agree with you about the
>>>>>>>> following point:
>>>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>>>> nftables, ... )
>>>>>>>> --this can reduce the load of compute node.
>>>>>>>>> * rpc communication mechanisms.
>>>>>>>> -- this can reduce the load of neutron server can you
>>>>>>>> help me to review my BP specs?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>>>>>>> mangelajo at redhat.com >
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi it's a very interesting topic, I was getting ready
>>>>>>>>> to raise the same concerns about our security groups
>>>>>>>>> implementation,
>>>>>>> shihanzhang
>>>>>>>>> thank you for starting this topic.
>>>>>>>>>
>>>>>>>>> Not only at low level where (with our default security
>>>>>>>>> group rules -allow all incoming from 'default' sg- the
>>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>>>>>>>>> "security_group_rules_for_devices" rpc call from
>>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>>>>>>>>>> 100MB,
>>>>>>>>> generating serious scalability issues or
>>>>>>>>> timeouts/retries that totally break neutron service.
>>>>>>>>>
>>>>>>>>> (example trace of that RPC call with a few instances
>>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>>>>>>>>>
>>>>>>>>> I believe that we also need to review the RPC calling
>>>>>>>>> mechanism for the OVS agent here, there are several
>>>>>>>>> possible approaches to
>>>>>>> breaking
>>>>>>>>> down (or/and CIDR compressing) the information we
>>>>>>>>> return via this
>>>>>>> api
>>>>>>>>> call.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So we have to look at two things here:
>>>>>>>>>
>>>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>>>> nftables, ... ) * rpc communication mechanisms.
>>>>>>>>>
>>>>>>>>> Best regards, Miguel Ángel.
>>>>>>>>>
>>>>>>>>> ----- Mensaje original -----
>>>>>>>>>
>>>>>>>>>> Do you though about nftables that will replace
>>>>>>> {ip,ip6,arp,eb}tables?
>>>>>>>>>> It also based on the rule set mechanism. The issue
>>>>>>>>>> in that proposition, it's only stable since the
>>>>>>>>>> begin
>>>>>>> of the
>>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>>>>>>>>>> I don't list here (leverage iptables
>>>>>>> limitation,
>>>>>>>>>> efficient update rule, rule set, standardization of
>>>>>>>>>> netfilter commands...).
>>>>>>>>>
>>>>>>>>>> Édouard.
>>>>>>>>>
>>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>>>>>>>>>> henry4hly at gmail.com > wrote:
>>>>>>>>>
>>>>>>>>>>> we have done some tests, but have different
>>>>>>>>>>> result: the
>>>>>>> performance is
>>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>>>>>>>>>>> but huge gap between enable/disable iptable hook
>>>>>>>>>>> on linux bridge
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>>>>>>> ayshihanzhang at 126.com
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> Now I have not get accurate test data, but I can
>>>>>>>>>>>> confirm the following points?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>>>>>>>>>>>> is liner,
>>>>>>> iptable
>>>>>>>>>>>> filter it one by one, if a VM in default
>>>>>>>>>>>> security group and this default security group
>>>>>>>>>>>> have many members, but ipset chain is set, the
>>>>>>>>>>>> time
>>>>>>> ipset
>>>>>>>>>>>> filter one and many member is not much
>>>>>>>>>>>> difference.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> 2. when the iptable rule is very large, the
>>>>>>>>>>>> probability of
>>>>>>> failure
>>>>>>>>>>>> that iptable-save save the iptable rule is very
>>>>>>>>>>>> large.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>>>>>>>>>>>> blak111 at gmail.com
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> This sounds like a good idea to handle some of
>>>>>>>>>>>>> the
>>>>>>> performance
>>>>>>>>>>>>> issues until the ovs firewall can be
>>>>>>>>>>>>> implemented down the the line.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Do you have any performance comparisons?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>>>>>>> ayshihanzhang at 126.com >
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> Now in neutron, it use iptable implementing
>>>>>>>>>>>>>> security
>>>>>>> group, but
>>>>>>>>>>>>>> the performance of this implementation is
>>>>>>>>>>>>>> very poor, there
>>>>>>> is a bug:
>>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>to
>>>>>>> reflect this
>>>>>>>>>>>>>> problem. In his test, w ith default security
>>>>>>>>>>>>>> groups(which has remote security group),
>>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>>>>>>>>>>>>>> Iptable rules
>>>>>>> on evry
>>>>>>>>>>>>>> compute node, although his patch can reduce
>>>>>>>>>>>>>> the processing time, but
>>>>>>> it don't
>>>>>>>>>>>>>> solve this problem fundamentally. I have
>>>>>>>>>>>>>> commit a BP to solve this
>>>>>>> problem:
>>>>>>>>>>>>>>
>>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>>>
>>>>>>>
>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> There are other people interested in this
>>>>>>>>>>>>>> it?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>>>>>
>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>
>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>
>>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>
>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> OpenStack-dev at lists.openstack.org >
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>>>>>>>
>>>>>>>>>
>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>>>>>>
>>>>>>>>
>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>>>>>>
>>>>>>>>
>>>
>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>>>>>
>>>>>>
>>>>>>
>>>
>>> _______________________________________________
>>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>
>>>>>>
>>_______________________________________________
>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>
>>>>>
>>>>>
>>>> _______________________________________________ OpenStack-dev
>>>> mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>_______________________________________________
>>>>>>OpenStack-dev
>>>>
>>mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>_______________________________________________
>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev
>>>>>
>>mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________ OpenStack-dev
>>> mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>-----BEGIN PGP SIGNATURE-----
>>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
>>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
>>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
>>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
>>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
>>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
>>=TxAT
>>-----END PGP SIGNATURE-----
>>
>>_______________________________________________
>>OpenStack-dev mailing list
>>OpenStack-dev at lists.openstack.org
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev






--
Best wishes!
Baohua


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140821/2cf16502/attachment-0001.html>

More information about the OpenStack-dev mailing list