Open Stack

+1 "NFTablesDriver"!

Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
https://home.regit.org/2014/02/suricata-and-nftables/

Then, I'm wondering here... What benefits might come for OpenStack Nova /
Neutron, if it comes with a NFTables driver, instead of the current
IPTables?!

* Efficient Security Group design?
* Better FWaaS, maybe with NAT(44/66) support?
* Native support for IPv6, with the defamed NAT66 built-in, simpler
"Floating IP" implementation, for both v4 and v6 networks under a single
implementation (*I don't like NAT66, I prefer a `routed Floating IPv6`
version*)?
* Metadata over IPv6 still using NAT(66) (*I don't like NAT66*), single
implementation?
* Suricata-as-a-Service?!

It sounds pretty cool!   :-)


On 20 August 2014 23:16, Baohua Yang <yangbaohua at gmail.com> wrote:

> Great！
> We met similar problems.
> The current mechanisms produce too many iptables rules, and it's hard to
> debug.
> Really look forward to seeing a more efficient security group design.
>
>
> On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <mestery at noironetworks.com>
> wrote:
>
>> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzhang at 126.com>
>> wrote:
>> >
>> > With the deployment 'nova + neutron + openvswitch', when we bulk create
>> > about 500 VM with a default security group, the CPU usage of
>> neutron-server
>> > and openvswitch agent is very high, especially the CPU usage of
>> openvswitch
>> > agent will be 100%, this will cause creating VMs failed.
>> >
>> > With the method discussed in mailist:
>> >
>> > 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
>> >
>> > 3) sg rpc optimization (with fanout)
>> > (https://review.openstack.org/#/c/104522/)
>> >
>> > I have implement  these two scheme in my deployment,  when we again bulk
>> > create about 500 VM with a default security group, the CPU usage of
>> > openvswitch agent will reduce to 10%, even lower than 10%, so I think
>> the
>> > iprovement of these two options are very efficient.
>> >
>> > Who can help us to review our spec?
>> >
>> This is great work! These are on my list of things to review in detail
>> soon, but given the Neutron sprint this week, I haven't had time yet.
>> I'll try to remedy that by the weekend.
>>
>> Thanks!
>> Kyle
>>
>> >    Best regards,
>> > shihanzhang
>> >
>> >
>> >
>> >
>> >
>> > At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
>> >>-----BEGIN PGP SIGNED MESSAGE-----
>> >>Hash: SHA512
>> >>
>> >>Oh, so you have the enhancement implemented? Great! Any numbers that
>> >>shows how much we gain from that?
>> >>
>> >>/Ihar
>> >>
>> >>On 03/07/14 02:49, shihanzhang wrote:
>> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>> >>> I will modify my spec, when the spec is approved, I will commit the
>> >>> codes as soon as possilbe!
>> >>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
>> >>> wrote:
>> >>>>
>> >>>> Nice Shihanzhang,
>> >>>>
>> >>>> Do you mean the ipset implementation is ready, or just the
>> >>>> spec?.
>> >>>>
>> >>>>
>> >>>> For the SG group refactor, I don't worry about who does it, or
>> >>>> who takes the credit, but I believe it's important we address
>> >>>> this bottleneck during Juno trying to match nova's scalability.
>> >>>>
>> >>>> Best regards, Miguel Ángel.
>> >>>>
>> >>>>
>> >>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>> >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>> >>>>> split  the work in several specs, I have finished the work (
>> >>>>> ipset optimization), you can do 'sg rpc optimization (without
>> >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>> >>>>> I think we need talk about it, because just using ipset to
>> >>>>> optimize security group agent codes does not bring the best
>> >>>>> results!
>> >>>>>
>> >>>>> Best regards, shihanzhang.
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
>> >>>>> wrote:
>> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>> >>>
>> >>>> Shihazhang,
>> >>>
>> >>>> I really believe we need the RPC refactor done for this cycle,
>> >>>> and given the close deadlines we have (July 10 for spec
>> >>>> submission and July 20 for spec approval).
>> >>>
>> >>>> Don't you think it's going to be better to split the work in
>> >>>> several specs?
>> >>>
>> >>>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>> >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>> >>>> , me)
>> >>>
>> >>>
>> >>>> This way we increase the chances of having part of this for the
>> >>>> Juno cycle. If we go for something too complicated is going to
>> >>>> take more time for approval.
>> >>>
>> >>>
>> >>> I agree. And it not only increases chances to get at least some of
>> >>> those highly demanded performance enhancements to get into Juno,
>> >>> it's also "the right thing to do" (c). It's counterproductive to
>> >>> put multiple vaguely related enhancements in single spec. This
>> >>> would dim review focus and put us into position of getting
>> >>> 'all-or-nothing'. We can't afford that.
>> >>>
>> >>> Let's leave one spec per enhancement. @Shihazhang, what do you
>> >>> think?
>> >>>
>> >>>
>> >>>> Also, I proposed the details of "2", trying to bring awareness
>> >>>> on the topic, as I have been working with the scale lab in Red
>> >>>> Hat to find and understand those issues, I have a very good
>> >>>> knowledge of the problem and I believe I could make a very fast
>> >>>> advance on the issue at the RPC level.
>> >>>
>> >>>> Given that, I'd like to work on this specific part, whether or
>> >>>> not we split the specs, as it's something we believe critical
>> >>>> for neutron scalability and thus, *nova parity*.
>> >>>
>> >>>> I will start a separate spec for "2", later on, if you find it
>> >>>> ok, we keep them as separate ones, if you believe having just 1
>> >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>> >>>> can incorporate my spec in yours, but then
>> >>>> "add-ipset-to-security" is not a good spec title to put all this
>> >>>> together.
>> >>>
>> >>>
>> >>>> Best regards, Miguel Ángel.
>> >>>
>> >>>
>> >>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>> >>>>>
>> >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>> >>>>> spes, but I will also optimization the RPC from security group
>> >>>>> agent to neutron server. Now the modle is
>> >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>> >>>>> sg2..]', this can reduce the size of RPC respose message from
>> >>>>> neutron server to security group agent.
>> >>>>>
>> >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>> >>>>> <mangelajo at redhat.com> wrote:
>> >>>>>>
>> >>>>>>
>> >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>> >>>>>> work into this problem, I could file an specific spec for
>> >>>>>> the security group RPC optimization, a masterplan in two
>> >>>>>> steps:
>> >>>>>>
>> >>>>>> 1) Refactor the current RPC communication for
>> >>>>>> security_groups_for_devices, which could be used for full
>> >>>>>> syncs, etc..
>> >>>>>>
>> >>>>>> 2) Benchmark && make use of a fanout queue per security
>> >>>>>> group to make sure only the hosts with instances on a
>> >>>>>> certain security group get the updates as they happen.
>> >>>>>>
>> >>>>>> @shihanzhang do you find it reasonable?
>> >>>>>>
>> >>>>>>
>> >>>>>>
>> >>>>>> ----- Original Message -----
>> >>>>>>> ----- Original Message -----
>> >>>>>>>> @Nachi: Yes that could a good improvement to factorize
>> >>>>>>>> the RPC
>> >>>>>>> mechanism.
>> >>>>>>>>
>> >>>>>>>> Another idea: What about creating a RPC topic per
>> >>>>>>>> security group (quid of the
>> >>>>>>> RPC topic
>> >>>>>>>> scalability) on which an agent subscribes if one of its
>> >>>>>>>> ports is
>> >>>>>>> associated
>> >>>>>>>> to the security group?
>> >>>>>>>>
>> >>>>>>>> Regards, Édouard.
>> >>>>>>>>
>> >>>>>>>>
>> >>>
>> >>>
>> >>>>>>> Hmm, Interesting,
>> >>>
>> >>>>>>> @Nachi, I'm not sure I fully understood:
>> >>>
>> >>>
>> >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>> >>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>> >>>
>> >>>
>> >>>>>>> Probably we may need to include also the SG_IP_LIST =
>> >>>>>>> [SG_IP1, SG_IP2] ...
>> >>>
>> >>>
>> >>>>>>> and let the agent do all the combination work.
>> >>>
>> >>>>>>> Something like this could make sense?
>> >>>
>> >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>> >>>>>>> SG2:{IPs:[....],RULES:[....]} }
>> >>>
>> >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>> >>>
>> >>>
>> >>>>>>> @Edouard, actually I like the idea of having the agent
>> >>>>>>> subscribed to security groups they have ports on... That
>> >>>>>>> would remove the need to include all the security groups
>> >>>>>>> information on every call...
>> >>>
>> >>>>>>> But would need another call to get the full information of
>> >>>>>>> a set of security groups at start/resync if we don't
>> >>>>>>> already have any.
>> >>>
>> >>>
>> >>>>>>>>
>> >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>> >>>>>>> ayshihanzhang at 126.com >
>> >>>>>>>> wrote:
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> hi Miguel Ángel, I am very agree with you about the
>> >>>>>>>> following point:
>> >>>>>>>>> * physical implementation on the hosts (ipsets,
>> >>>>>>>>> nftables, ... )
>> >>>>>>>> --this can reduce the load of compute node.
>> >>>>>>>>> * rpc communication mechanisms.
>> >>>>>>>> -- this can reduce the load of neutron server can you
>> >>>>>>>> help me to review my BP specs?
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>> >>>>>>> mangelajo at redhat.com >
>> >>>>>>>> wrote:
>> >>>>>>>>>
>> >>>>>>>>> Hi it's a very interesting topic, I was getting ready
>> >>>>>>>>> to raise the same concerns about our security groups
>> >>>>>>>>> implementation,
>> >>>>>>> shihanzhang
>> >>>>>>>>> thank you for starting this topic.
>> >>>>>>>>>
>> >>>>>>>>> Not only at low level where (with our default security
>> >>>>>>>>> group rules -allow all incoming from 'default' sg- the
>> >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>> >>>>>>>>> "security_group_rules_for_devices" rpc call from
>> >>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>> >>>>>>>>>> 100MB,
>> >>>>>>>>> generating serious scalability issues or
>> >>>>>>>>> timeouts/retries that totally break neutron service.
>> >>>>>>>>>
>> >>>>>>>>> (example trace of that RPC call with a few instances
>> >>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>> >>>>>>>>>
>> >>>>>>>>> I believe that we also need to review the RPC calling
>> >>>>>>>>> mechanism for the OVS agent here, there are several
>> >>>>>>>>> possible approaches to
>> >>>>>>> breaking
>> >>>>>>>>> down (or/and CIDR compressing) the information we
>> >>>>>>>>> return via this
>> >>>>>>> api
>> >>>>>>>>> call.
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>> So we have to look at two things here:
>> >>>>>>>>>
>> >>>>>>>>> * physical implementation on the hosts (ipsets,
>> >>>>>>>>> nftables, ... ) * rpc communication mechanisms.
>> >>>>>>>>>
>> >>>>>>>>> Best regards, Miguel Ángel.
>> >>>>>>>>>
>> >>>>>>>>> ----- Mensaje original -----
>> >>>>>>>>>
>> >>>>>>>>>> Do you though about nftables that will replace
>> >>>>>>> {ip,ip6,arp,eb}tables?
>> >>>>>>>>>> It also based on the rule set mechanism. The issue
>> >>>>>>>>>> in that proposition, it's only stable since the
>> >>>>>>>>>> begin
>> >>>>>>> of the
>> >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>> >>>>>>>>>> I don't list here (leverage iptables
>> >>>>>>> limitation,
>> >>>>>>>>>> efficient update rule, rule set, standardization of
>> >>>>>>>>>> netfilter commands...).
>> >>>>>>>>>
>> >>>>>>>>>> Édouard.
>> >>>>>>>>>
>> >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>> >>>>>>>>>> henry4hly at gmail.com > wrote:
>> >>>>>>>>>
>> >>>>>>>>>>> we have done some tests, but have different
>> >>>>>>>>>>> result: the
>> >>>>>>> performance is
>> >>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>> >>>>>>>>>>> but huge gap between enable/disable iptable hook
>> >>>>>>>>>>> on linux bridge
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>> >>>>>>> ayshihanzhang at 126.com
>> >>>>>>>>>>>>
>> >>>>>>>>>>> wrote:
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>> Now I have not get accurate test data, but I can
>> >>>>>>>>>>>> confirm the following points?
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>> >>>>>>>>>>>> is liner,
>> >>>>>>> iptable
>> >>>>>>>>>>>> filter it one by one, if a VM in default
>> >>>>>>>>>>>> security group and this default security group
>> >>>>>>>>>>>> have many members, but ipset chain is set, the
>> >>>>>>>>>>>> time
>> >>>>>>> ipset
>> >>>>>>>>>>>> filter one and many member is not much
>> >>>>>>>>>>>> difference.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>> 2. when the iptable rule is very large, the
>> >>>>>>>>>>>> probability of
>> >>>>>>> failure
>> >>>>>>>>>>>> that iptable-save save the iptable rule is very
>> >>>>>>>>>>>> large.
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>> >>>>>>>>>>>> blak111 at gmail.com
>> >>>>>>>> wrote:
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>> This sounds like a good idea to handle some of
>> >>>>>>>>>>>>> the
>> >>>>>>> performance
>> >>>>>>>>>>>>> issues until the ovs firewall can be
>> >>>>>>>>>>>>> implemented down the the line.
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>>> Do you have any performance comparisons?
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>> >>>>>>> ayshihanzhang at 126.com >
>> >>>>>>>>>>>>> wrote:
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>>> Hello all,
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>>> Now in neutron, it use iptable implementing
>> >>>>>>>>>>>>>> security
>> >>>>>>> group, but
>> >>>>>>>>>>>>>> the performance of this implementation is
>> >>>>>>>>>>>>>> very poor, there
>> >>>>>>> is a bug:
>> >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>to
>> >>>>>>> reflect this
>> >>>>>>>>>>>>>> problem. In his test, w ith default security
>> >>>>>>>>>>>>>> groups(which has remote security group),
>> >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>> >>>>>>>>>>>>>> Iptable rules
>> >>>>>>> on evry
>> >>>>>>>>>>>>>> compute node, although his patch can reduce
>> >>>>>>>>>>>>>> the processing time, but
>> >>>>>>> it don't
>> >>>>>>>>>>>>>> solve this problem fundamentally. I have
>> >>>>>>>>>>>>>> commit a BP to solve this
>> >>>>>>> problem:
>> >>>>>>>>>>>>>>
>> >>>>>>>
>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>> >>>
>> >>>>>>>
>> >>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>>>> There are other people interested in this
>> >>>>>>>>>>>>>> it?
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>>> _______________________________________________
>> >>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>>>>
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>>>>>
>> >>>
>> >>>>>>>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>> >>>>>>>>>>
>> >>>>>>>>>>>>
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>>
>> >>>>>>>
>> >>>>
>> >>>>>>>>>>>>>
>> >>>>>>>>>>
>> >>>>>>>>>
>> >>>>>>>>>>> _______________________________________________
>> >>>>>>>>>>
>> >>>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>>
>> >>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>> >>>>>>>>>>>
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>>
>> >>>>>>>
>> >>>>
>> >>>>>>>>>
>> >>>>>>>>>> _______________________________________________
>> >>>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>> >>>>>>>>>>
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>>
>> >>>>>>>
>> >>>
>> >>>>>>>>> _______________________________________________
>> >>>>>>>>> OpenStack-dev mailing list
>> >>>>>>>>> OpenStack-dev at lists.openstack.org >
>> >>>>>>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>>>>>>>
>> >>>>>>>>>
>> >>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> _______________________________________________
>> >>>>>>>> OpenStack-dev mailing list
>> >>>>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>>>>>> _______________________________________________
>> >>>>>>>> OpenStack-dev mailing list
>> >>>>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>>>>>>
>> >>>>>>>>
>> >>>
>> >>>
>> >>>>>>> _______________________________________________
>> >>>>>>> OpenStack-dev mailing list
>> >>>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>>>>>
>> >>>>>>
>> >>>>>>
>> >>>
>> >>> _______________________________________________
>> >>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>>
>> >>>>>>
>> >>_______________________________________________
>> >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>
>> >>>
>> >>>>>
>> >>>>>
>> >>>> _______________________________________________ OpenStack-dev
>> >>>> mailing list OpenStack-dev at lists.openstack.org
>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>
>> >>>>>>_______________________________________________
>> >>>>>>OpenStack-dev
>> >>>>
>> >>mailing list
>> >>>>>> OpenStack-dev at lists.openstack.org
>> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>>
>> >>_______________________________________________
>> >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>
>> >>>>
>> >>>>_______________________________________________
>> >>>>OpenStack-dev
>> >>>>>
>> >>mailing list
>> >>>> OpenStack-dev at lists.openstack.org
>> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________ OpenStack-dev
>> >>> mailing list OpenStack-dev at lists.openstack.org
>> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>
>> >>-----BEGIN PGP SIGNATURE-----
>> >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>> >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>> >>
>> >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
>> >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
>> >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
>> >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
>> >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
>> >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
>> >>=TxAT
>> >>-----END PGP SIGNATURE-----
>> >>
>> >>_______________________________________________
>> >>OpenStack-dev mailing list
>> >>OpenStack-dev at lists.openstack.org
>> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
>
> --
> Best wishes!
> Baohua
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140820/1e930044/attachment.html>