Open Stack

This is great work! Looking forward to seeing this get reviewed and
merged in Juno!

Kyle

On Thu, Aug 21, 2014 at 6:49 AM, Édouard Thuleau <thuleau at gmail.com> wrote:
> Nice job! That's awesome.
>
> Thanks,
> Édouard.
>
>
> On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo
> <mangelajo at redhat.com> wrote:
>>
>> Thank you shihanzhang!,
>>
>> I can't believe I didn't realize the ipset part spec was accepted I live
>> on my own bubble... I will be reviewing and testing/helping on that part
>> too during the next few days,  I was too concentrated in the RPC part.
>>
>>
>> Best regards,
>>
>> ----- Original Message -----
>> > hi neutroner!
>> > my patch about BP:
>> >
>> > https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security
>> > need install ipset in devstack, I have commit the patch:
>> > https://review.openstack.org/#/c/113453/, who can help me review it,
>> > thanks
>> > very much!
>> >
>> > Best regards,
>> > shihanzhang
>> >
>> >
>> >
>> >
>> > At 2014-08-21 10:47:59, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com>
>> > wrote:
>> >
>> >
>> >
>> > +1 "NFTablesDriver"!
>> >
>> > Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
>> > https://home.regit.org/2014/02/suricata-and-nftables/
>> >
>> > Then, I'm wondering here... What benefits might come for OpenStack Nova
>> > /
>> > Neutron, if it comes with a NFTables driver, instead of the current
>> > IPTables?!
>> >
>> > * E fficient Security Group design?
>> > * Better FWaaS, maybe with NAT(44/66) support?
>> > * Native support for IPv6, with the defamed NAT66 built-in, simpler
>> > "Floating
>> > IP" implementation, for both v4 and v6 networks under a single
>> > implementation ( I don't like NAT66, I prefer a `routed Floating IPv6`
>> > version ) ?
>> > * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single
>> > implementation?
>> > * Suricata-as-a-Service?!
>> >
>> > It sounds pretty cool! :-)
>> >
>> >
>> > On 20 August 2014 23:16, Baohua Yang < yangbaohua at gmail.com > wrote:
>> >
>> >
>> >
>> > Great！
>> > We met similar problems.
>> > The current mechanisms produce too many iptables rules, and it's hard to
>> > debug.
>> > Really look forward to seeing a more efficient security group design.
>> >
>> >
>> > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <
>> > mestery at noironetworks.com >
>> > wrote:
>> >
>> >
>> >
>> > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < ayshihanzhang at 126.com >
>> > wrote:
>> > >
>> > > With the deployment 'nova + neutron + openvswitch', when we bulk
>> > > create
>> > > about 500 VM with a default security group, the CPU usage of
>> > > neutron-server
>> > > and openvswitch agent is very high, especially the CPU usage of
>> > > openvswitch
>> > > agent will be 100%, this will cause creating VMs failed.
>> > >
>> > > With the method discussed in mailist:
>> > >
>> > > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ )
>> > >
>> > > 3) sg rpc optimization (with fanout)
>> > > ( https://review.openstack.org/#/c/104522/ )
>> > >
>> > > I have implement these two scheme in my deployment, when we again bulk
>> > > create about 500 VM with a default security group, the CPU usage of
>> > > openvswitch agent will reduce to 10%, even lower than 10%, so I think
>> > > the
>> > > iprovement of these two options are very efficient.
>> > >
>> > > Who can help us to review our spec?
>> > >
>> > This is great work! These are on my list of things to review in detail
>> > soon, but given the Neutron sprint this week, I haven't had time yet.
>> > I'll try to remedy that by the weekend.
>> >
>> > Thanks!
>> > Kyle
>> >
>> > > Best regards,
>> > > shihanzhang
>> > >
>> > >
>> > >
>> > >
>> > >
>> > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < ihrachys at redhat.com >
>> > > wrote:
>> > >>-----BEGIN PGP SIGNED MESSAGE-----
>> > >>Hash: SHA512
>> > >>
>> > >>Oh, so you have the enhancement implemented? Great! Any numbers that
>> > >>shows how much we gain from that?
>> > >>
>> > >>/Ihar
>> > >>
>> > >>On 03/07/14 02:49, shihanzhang wrote:
>> > >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>> > >>> I will modify my spec, when the spec is approved, I will commit the
>> > >>> codes as soon as possilbe!
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < majopela at redhat.com >
>> > >>> wrote:
>> > >>>>
>> > >>>> Nice Shihanzhang,
>> > >>>>
>> > >>>> Do you mean the ipset implementation is ready, or just the
>> > >>>> spec?.
>> > >>>>
>> > >>>>
>> > >>>> For the SG group refactor, I don't worry about who does it, or
>> > >>>> who takes the credit, but I believe it's important we address
>> > >>>> this bottleneck during Juno trying to match nova's scalability.
>> > >>>>
>> > >>>> Best regards, Miguel Ángel.
>> > >>>>
>> > >>>>
>> > >>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>> > >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>> > >>>>> split the work in several specs, I have finished the work (
>> > >>>>> ipset optimization), you can do 'sg rpc optimization (without
>> > >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>> > >>>>> I think we need talk about it, because just using ipset to
>> > >>>>> optimize security group agent codes does not bring the best
>> > >>>>> results!
>> > >>>>>
>> > >>>>> Best regards, shihanzhang.
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" < ihrachys at redhat.com >
>> > >>>>> wrote:
>> > >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>> > >>>
>> > >>>> Shihazhang,
>> > >>>
>> > >>>> I really believe we need the RPC refactor done for this cycle,
>> > >>>> and given the close deadlines we have (July 10 for spec
>> > >>>> submission and July 20 for spec approval).
>> > >>>
>> > >>>> Don't you think it's going to be better to split the work in
>> > >>>> several specs?
>> > >>>
>> > >>>> 1) ipset optimization (you) 2) sg rpc optimization (without
>> > >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>> > >>>> , me)
>> > >>>
>> > >>>
>> > >>>> This way we increase the chances of having part of this for the
>> > >>>> Juno cycle. If we go for something too complicated is going to
>> > >>>> take more time for approval.
>> > >>>
>> > >>>
>> > >>> I agree. And it not only increases chances to get at least some of
>> > >>> those highly demanded performance enhancements to get into Juno,
>> > >>> it's also "the right thing to do" (c). It's counterproductive to
>> > >>> put multiple vaguely related enhancements in single spec. This
>> > >>> would dim review focus and put us into position of getting
>> > >>> 'all-or-nothing'. We can't afford that.
>> > >>>
>> > >>> Let's leave one spec per enhancement. @Shihazhang, what do you
>> > >>> think?
>> > >>>
>> > >>>
>> > >>>> Also, I proposed the details of "2", trying to bring awareness
>> > >>>> on the topic, as I have been working with the scale lab in Red
>> > >>>> Hat to find and understand those issues, I have a very good
>> > >>>> knowledge of the problem and I believe I could make a very fast
>> > >>>> advance on the issue at the RPC level.
>> > >>>
>> > >>>> Given that, I'd like to work on this specific part, whether or
>> > >>>> not we split the specs, as it's something we believe critical
>> > >>>> for neutron scalability and thus, *nova parity*.
>> > >>>
>> > >>>> I will start a separate spec for "2", later on, if you find it
>> > >>>> ok, we keep them as separate ones, if you believe having just 1
>> > >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>> > >>>> can incorporate my spec in yours, but then
>> > >>>> "add-ipset-to-security" is not a good spec title to put all this
>> > >>>> together.
>> > >>>
>> > >>>
>> > >>>> Best regards, Miguel Ángel.
>> > >>>
>> > >>>
>> > >>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>> > >>>>>
>> > >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>> > >>>>> spes, but I will also optimization the RPC from security group
>> > >>>>> agent to neutron server. Now the modle is
>> > >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>> > >>>>> sg2..]', this can reduce the size of RPC respose message from
>> > >>>>> neutron server to security group agent.
>> > >>>>>
>> > >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>> > >>>>> < mangelajo at redhat.com > wrote:
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>> > >>>>>> work into this problem, I could file an specific spec for
>> > >>>>>> the security group RPC optimization, a masterplan in two
>> > >>>>>> steps:
>> > >>>>>>
>> > >>>>>> 1) Refactor the current RPC communication for
>> > >>>>>> security_groups_for_devices, which could be used for full
>> > >>>>>> syncs, etc..
>> > >>>>>>
>> > >>>>>> 2) Benchmark && make use of a fanout queue per security
>> > >>>>>> group to make sure only the hosts with instances on a
>> > >>>>>> certain security group get the updates as they happen.
>> > >>>>>>
>> > >>>>>> @shihanzhang do you find it reasonable?
>> > >>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>>>> ----- Original Message -----
>> > >>>>>>> ----- Original Message -----
>> > >>>>>>>> @Nachi: Yes that could a good improvement to factorize
>> > >>>>>>>> the RPC
>> > >>>>>>> mechanism.
>> > >>>>>>>>
>> > >>>>>>>> Another idea: What about creating a RPC topic per
>> > >>>>>>>> security group (quid of the
>> > >>>>>>> RPC topic
>> > >>>>>>>> scalability) on which an agent subscribes if one of its
>> > >>>>>>>> ports is
>> > >>>>>>> associated
>> > >>>>>>>> to the security group?
>> > >>>>>>>>
>> > >>>>>>>> Regards, Édouard.
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>
>> > >>>
>> > >>>>>>> Hmm, Interesting,
>> > >>>
>> > >>>>>>> @Nachi, I'm not sure I fully understood:
>> > >>>
>> > >>>
>> > >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>> > >>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>> > >>>
>> > >>>
>> > >>>>>>> Probably we may need to include also the SG_IP_LIST =
>> > >>>>>>> [SG_IP1, SG_IP2] ...
>> > >>>
>> > >>>
>> > >>>>>>> and let the agent do all the combination work.
>> > >>>
>> > >>>>>>> Something like this could make sense?
>> > >>>
>> > >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>> > >>>>>>> SG2:{IPs:[....],RULES:[....]} }
>> > >>>
>> > >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>> > >>>
>> > >>>
>> > >>>>>>> @Edouard, actually I like the idea of having the agent
>> > >>>>>>> subscribed to security groups they have ports on... That
>> > >>>>>>> would remove the need to include all the security groups
>> > >>>>>>> information on every call...
>> > >>>
>> > >>>>>>> But would need another call to get the full information of
>> > >>>>>>> a set of security groups at start/resync if we don't
>> > >>>>>>> already have any.
>> > >>>
>> > >>>
>> > >>>>>>>>
>> > >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>> > >>>>>>> ayshihanzhang at 126.com >
>> > >>>>>>>> wrote:
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> hi Miguel Ángel, I am very agree with you about the
>> > >>>>>>>> following point:
>> > >>>>>>>>> * physical implementation on the hosts (ipsets,
>> > >>>>>>>>> nftables, ... )
>> > >>>>>>>> --this can reduce the load of compute node.
>> > >>>>>>>>> * rpc communication mechanisms.
>> > >>>>>>>> -- this can reduce the load of neutron server can you
>> > >>>>>>>> help me to review my BP specs?
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>> > >>>>>>> mangelajo at redhat.com >
>> > >>>>>>>> wrote:
>> > >>>>>>>>>
>> > >>>>>>>>> Hi it's a very interesting topic, I was getting ready
>> > >>>>>>>>> to raise the same concerns about our security groups
>> > >>>>>>>>> implementation,
>> > >>>>>>> shihanzhang
>> > >>>>>>>>> thank you for starting this topic.
>> > >>>>>>>>>
>> > >>>>>>>>> Not only at low level where (with our default security
>> > >>>>>>>>> group rules -allow all incoming from 'default' sg- the
>> > >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>> > >>>>>>>>> "security_group_rules_for_devices" rpc call from
>> > >>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>> > >>>>>>>>>> 100MB,
>> > >>>>>>>>> generating serious scalability issues or
>> > >>>>>>>>> timeouts/retries that totally break neutron service.
>> > >>>>>>>>>
>> > >>>>>>>>> (example trace of that RPC call with a few instances
>> > >>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>> > >>>>>>>>>
>> > >>>>>>>>> I believe that we also need to review the RPC calling
>> > >>>>>>>>> mechanism for the OVS agent here, there are several
>> > >>>>>>>>> possible approaches to
>> > >>>>>>> breaking
>> > >>>>>>>>> down (or/and CIDR compressing) the information we
>> > >>>>>>>>> return via this
>> > >>>>>>> api
>> > >>>>>>>>> call.
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>> So we have to look at two things here:
>> > >>>>>>>>>
>> > >>>>>>>>> * physical implementation on the hosts (ipsets,
>> > >>>>>>>>> nftables, ... ) * rpc communication mechanisms.
>> > >>>>>>>>>
>> > >>>>>>>>> Best regards, Miguel Ángel.
>> > >>>>>>>>>
>> > >>>>>>>>> ----- Mensaje original -----
>> > >>>>>>>>>
>> > >>>>>>>>>> Do you though about nftables that will replace
>> > >>>>>>> {ip,ip6,arp,eb}tables?
>> > >>>>>>>>>> It also based on the rule set mechanism. The issue
>> > >>>>>>>>>> in that proposition, it's only stable since the
>> > >>>>>>>>>> begin
>> > >>>>>>> of the
>> > >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>> > >>>>>>>>>> I don't list here (leverage iptables
>> > >>>>>>> limitation,
>> > >>>>>>>>>> efficient update rule, rule set, standardization of
>> > >>>>>>>>>> netfilter commands...).
>> > >>>>>>>>>
>> > >>>>>>>>>> Édouard.
>> > >>>>>>>>>
>> > >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>> > >>>>>>>>>> henry4hly at gmail.com > wrote:
>> > >>>>>>>>>
>> > >>>>>>>>>>> we have done some tests, but have different
>> > >>>>>>>>>>> result: the
>> > >>>>>>> performance is
>> > >>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>> > >>>>>>>>>>> but huge gap between enable/disable iptable hook
>> > >>>>>>>>>>> on linux bridge
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>> > >>>>>>> ayshihanzhang at 126.com
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>> wrote:
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>> Now I have not get accurate test data, but I can
>> > >>>>>>>>>>>> confirm the following points?
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>> > >>>>>>>>>>>> is liner,
>> > >>>>>>> iptable
>> > >>>>>>>>>>>> filter it one by one, if a VM in default
>> > >>>>>>>>>>>> security group and this default security group
>> > >>>>>>>>>>>> have many members, but ipset chain is set, the
>> > >>>>>>>>>>>> time
>> > >>>>>>> ipset
>> > >>>>>>>>>>>> filter one and many member is not much
>> > >>>>>>>>>>>> difference.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>> 2. when the iptable rule is very large, the
>> > >>>>>>>>>>>> probability of
>> > >>>>>>> failure
>> > >>>>>>>>>>>> that iptable-save save the iptable rule is very
>> > >>>>>>>>>>>> large.
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>> > >>>>>>>>>>>> blak111 at gmail.com
>> > >>>>>>>> wrote:
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>>> This sounds like a good idea to handle some of
>> > >>>>>>>>>>>>> the
>> > >>>>>>> performance
>> > >>>>>>>>>>>>> issues until the ovs firewall can be
>> > >>>>>>>>>>>>> implemented down the the line.
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>> Do you have any performance comparisons?
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>> > >>>>>>> ayshihanzhang at 126.com >
>> > >>>>>>>>>>>>> wrote:
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>>>> Hello all,
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>>>> Now in neutron, it use iptable implementing
>> > >>>>>>>>>>>>>> security
>> > >>>>>>> group, but
>> > >>>>>>>>>>>>>> the performance of this implementation is
>> > >>>>>>>>>>>>>> very poor, there
>> > >>>>>>> is a bug:
>> > >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>to
>> > >>>>>>> reflect this
>> > >>>>>>>>>>>>>> problem. In his test, w ith default security
>> > >>>>>>>>>>>>>> groups(which has remote security group),
>> > >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>> > >>>>>>>>>>>>>> Iptable rules
>> > >>>>>>> on evry
>> > >>>>>>>>>>>>>> compute node, although his patch can reduce
>> > >>>>>>>>>>>>>> the processing time, but
>> > >>>>>>> it don't
>> > >>>>>>>>>>>>>> solve this problem fundamentally. I have
>> > >>>>>>>>>>>>>> commit a BP to solve this
>> > >>>>>>> problem:
>> > >>>>>>>>>>>>>>
>> > >>>>>>>
>> > >>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>> > >>>
>> > >>>>>>>
>> > >>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>>> There are other people interested in this
>> > >>>>>>>>>>>>>> it?
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>>>> _______________________________________________
>> > >>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>>>
>> > >>>>>>>
>> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>>>>>>
>> > >>>
>> > >>>>>>>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>>> _______________________________________________
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>> > >>>>>>>>>>
>> > >>>>>>>>>>>>
>> > >>>>>>>
>> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>
>> > >>>>>>>>>>>>>
>> > >>>>>>>>>>
>> > >>>>>>>>>
>> > >>>>>>>>>>> _______________________________________________
>> > >>>>>>>>>>
>> > >>>>>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>>>>
>> > >>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>> > >>>>>>>>>>>
>> > >>>>>>>
>> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>>>
>> > >>>>>>>
>> > >>>>
>> > >>>>>>>>>
>> > >>>>>>>>>> _______________________________________________
>> > >>>>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>> > >>>>>>>>>>
>> > >>>>>>>
>> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>>>
>> > >>>>>>>
>> > >>>
>> > >>>>>>>>> _______________________________________________
>> > >>>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>>> OpenStack-dev at lists.openstack.org >
>> > >>>>>>>>>
>> > >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>>>>>>>>
>> > >>>>>>>>>
>> > >>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> _______________________________________________
>> > >>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>> OpenStack-dev at lists.openstack.org
>> > >>>>>>>>
>> > >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>>>>>> _______________________________________________
>> > >>>>>>>> OpenStack-dev mailing list
>> > >>>>>>>> OpenStack-dev at lists.openstack.org
>> > >>>>>>>>
>> > >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>>>>>>>
>> > >>>>>>>>
>> > >>>
>> > >>>
>> > >>>>>>> _______________________________________________
>> > >>>>>>> OpenStack-dev mailing list
>> > >>>>>>> OpenStack-dev at lists.openstack.org
>> > >>>>>>>
>> > >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>>>>>>
>> > >>>>>>
>> > >>>>>>
>> > >>>
>> > >>> _______________________________________________
>> > >>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>>
>> > >>>
>> > >>>>>>
>> > >>_______________________________________________
>> > >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>
>> > >>>
>> > >>>>>
>> > >>>>>
>> > >>>> _______________________________________________ OpenStack-dev
>> > >>>> mailing list OpenStack-dev at lists.openstack.org
>> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>>
>> > >>>>>>_______________________________________________
>> > >>>>>>OpenStack-dev
>> > >>>>
>> > >>mailing list
>> > >>>>>> OpenStack-dev at lists.openstack.org
>> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>
>> > >>>>>>
>> > >>_______________________________________________
>> > >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>> > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>>>
>> > >>>>
>> > >>>>_______________________________________________
>> > >>>>OpenStack-dev
>> > >>>>>
>> > >>mailing list
>> > >>>> OpenStack-dev at lists.openstack.org
>> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>>>
>> > >>>
>> > >>>
>> > >>>
>> > >>>
>> > >>> _______________________________________________ OpenStack-dev
>> > >>> mailing list OpenStack-dev at lists.openstack.org
>> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >>>
>> > >>-----BEGIN PGP SIGNATURE-----
>> > >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>> > >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>> > >>
>> > >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
>> > >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
>> > >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
>> > >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
>> > >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
>> > >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
>> > >>=TxAT
>> > >>-----END PGP SIGNATURE-----
>> > >>
>> > >>_______________________________________________
>> > >>OpenStack-dev mailing list
>> > >> OpenStack-dev at lists.openstack.org
>> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> > >
>> > >
>> > >
>> > > _______________________________________________
>> > > OpenStack-dev mailing list
>> > > OpenStack-dev at lists.openstack.org
>> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> > >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> > --
>> > Best wishes!
>> > Baohua
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>