Open Stack

Nice job! That's awesome.

Thanks,
Édouard.


On Thu, Aug 21, 2014 at 8:02 AM, Miguel Angel Ajo Pelayo <
mangelajo at redhat.com> wrote:

> Thank you shihanzhang!,
>
> I can't believe I didn't realize the ipset part spec was accepted I live
> on my own bubble... I will be reviewing and testing/helping on that part
> too during the next few days,  I was too concentrated in the RPC part.
>
>
> Best regards,
>
> ----- Original Message -----
> > hi neutroner!
> > my patch about BP:
> >
> https://blueprints.launchpad.net/openstack/?searchtext=add-ipset-to-security
> > need install ipset in devstack, I have commit the patch:
> > https://review.openstack.org/#/c/113453/, who can help me review it,
> thanks
> > very much!
> >
> > Best regards,
> > shihanzhang
> >
> >
> >
> >
> > At 2014-08-21 10:47:59, "Martinx - ジェームズ" <thiagocmartinsc at gmail.com>
> wrote:
> >
> >
> >
> > +1 "NFTablesDriver"!
> >
> > Also, NFTables, AFAIK, improves IDS systems, like Suricata, for example:
> > https://home.regit.org/2014/02/suricata-and-nftables/
> >
> > Then, I'm wondering here... What benefits might come for OpenStack Nova /
> > Neutron, if it comes with a NFTables driver, instead of the current
> > IPTables?!
> >
> > * E fficient Security Group design?
> > * Better FWaaS, maybe with NAT(44/66) support?
> > * Native support for IPv6, with the defamed NAT66 built-in, simpler
> "Floating
> > IP" implementation, for both v4 and v6 networks under a single
> > implementation ( I don't like NAT66, I prefer a `routed Floating IPv6`
> > version ) ?
> > * Metadata over IPv6 still using NAT(66) ( I don't like NAT66 ), single
> > implementation?
> > * Suricata-as-a-Service?!
> >
> > It sounds pretty cool! :-)
> >
> >
> > On 20 August 2014 23:16, Baohua Yang < yangbaohua at gmail.com > wrote:
> >
> >
> >
> > Great！
> > We met similar problems.
> > The current mechanisms produce too many iptables rules, and it's hard to
> > debug.
> > Really look forward to seeing a more efficient security group design.
> >
> >
> > On Thu, Jul 10, 2014 at 11:44 PM, Kyle Mestery <
> mestery at noironetworks.com >
> > wrote:
> >
> >
> >
> > On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang < ayshihanzhang at 126.com >
> wrote:
> > >
> > > With the deployment 'nova + neutron + openvswitch', when we bulk create
> > > about 500 VM with a default security group, the CPU usage of
> neutron-server
> > > and openvswitch agent is very high, especially the CPU usage of
> openvswitch
> > > agent will be 100%, this will cause creating VMs failed.
> > >
> > > With the method discussed in mailist:
> > >
> > > 1) ipset optimization ( https://review.openstack.org/#/c/100761/ )
> > >
> > > 3) sg rpc optimization (with fanout)
> > > ( https://review.openstack.org/#/c/104522/ )
> > >
> > > I have implement these two scheme in my deployment, when we again bulk
> > > create about 500 VM with a default security group, the CPU usage of
> > > openvswitch agent will reduce to 10%, even lower than 10%, so I think
> the
> > > iprovement of these two options are very efficient.
> > >
> > > Who can help us to review our spec?
> > >
> > This is great work! These are on my list of things to review in detail
> > soon, but given the Neutron sprint this week, I haven't had time yet.
> > I'll try to remedy that by the weekend.
> >
> > Thanks!
> > Kyle
> >
> > > Best regards,
> > > shihanzhang
> > >
> > >
> > >
> > >
> > >
> > > At 2014-07-03 10:08:21, "Ihar Hrachyshka" < ihrachys at redhat.com >
> wrote:
> > >>-----BEGIN PGP SIGNED MESSAGE-----
> > >>Hash: SHA512
> > >>
> > >>Oh, so you have the enhancement implemented? Great! Any numbers that
> > >>shows how much we gain from that?
> > >>
> > >>/Ihar
> > >>
> > >>On 03/07/14 02:49, shihanzhang wrote:
> > >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> > >>> I will modify my spec, when the spec is approved, I will commit the
> > >>> codes as soon as possilbe!
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" < majopela at redhat.com >
> > >>> wrote:
> > >>>>
> > >>>> Nice Shihanzhang,
> > >>>>
> > >>>> Do you mean the ipset implementation is ready, or just the
> > >>>> spec?.
> > >>>>
> > >>>>
> > >>>> For the SG group refactor, I don't worry about who does it, or
> > >>>> who takes the credit, but I believe it's important we address
> > >>>> this bottleneck during Juno trying to match nova's scalability.
> > >>>>
> > >>>> Best regards, Miguel Ángel.
> > >>>>
> > >>>>
> > >>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
> > >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> > >>>>> split the work in several specs, I have finished the work (
> > >>>>> ipset optimization), you can do 'sg rpc optimization (without
> > >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
> > >>>>> I think we need talk about it, because just using ipset to
> > >>>>> optimize security group agent codes does not bring the best
> > >>>>> results!
> > >>>>>
> > >>>>> Best regards, shihanzhang.
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" < ihrachys at redhat.com >
> > >>>>> wrote:
> > >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> > >>>
> > >>>> Shihazhang,
> > >>>
> > >>>> I really believe we need the RPC refactor done for this cycle,
> > >>>> and given the close deadlines we have (July 10 for spec
> > >>>> submission and July 20 for spec approval).
> > >>>
> > >>>> Don't you think it's going to be better to split the work in
> > >>>> several specs?
> > >>>
> > >>>> 1) ipset optimization (you) 2) sg rpc optimization (without
> > >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
> > >>>> , me)
> > >>>
> > >>>
> > >>>> This way we increase the chances of having part of this for the
> > >>>> Juno cycle. If we go for something too complicated is going to
> > >>>> take more time for approval.
> > >>>
> > >>>
> > >>> I agree. And it not only increases chances to get at least some of
> > >>> those highly demanded performance enhancements to get into Juno,
> > >>> it's also "the right thing to do" (c). It's counterproductive to
> > >>> put multiple vaguely related enhancements in single spec. This
> > >>> would dim review focus and put us into position of getting
> > >>> 'all-or-nothing'. We can't afford that.
> > >>>
> > >>> Let's leave one spec per enhancement. @Shihazhang, what do you
> > >>> think?
> > >>>
> > >>>
> > >>>> Also, I proposed the details of "2", trying to bring awareness
> > >>>> on the topic, as I have been working with the scale lab in Red
> > >>>> Hat to find and understand those issues, I have a very good
> > >>>> knowledge of the problem and I believe I could make a very fast
> > >>>> advance on the issue at the RPC level.
> > >>>
> > >>>> Given that, I'd like to work on this specific part, whether or
> > >>>> not we split the specs, as it's something we believe critical
> > >>>> for neutron scalability and thus, *nova parity*.
> > >>>
> > >>>> I will start a separate spec for "2", later on, if you find it
> > >>>> ok, we keep them as separate ones, if you believe having just 1
> > >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
> > >>>> can incorporate my spec in yours, but then
> > >>>> "add-ipset-to-security" is not a good spec title to put all this
> > >>>> together.
> > >>>
> > >>>
> > >>>> Best regards, Miguel Ángel.
> > >>>
> > >>>
> > >>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
> > >>>>>
> > >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
> > >>>>> spes, but I will also optimization the RPC from security group
> > >>>>> agent to neutron server. Now the modle is
> > >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
> > >>>>> sg2..]', this can reduce the size of RPC respose message from
> > >>>>> neutron server to security group agent.
> > >>>>>
> > >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
> > >>>>> < mangelajo at redhat.com > wrote:
> > >>>>>>
> > >>>>>>
> > >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
> > >>>>>> work into this problem, I could file an specific spec for
> > >>>>>> the security group RPC optimization, a masterplan in two
> > >>>>>> steps:
> > >>>>>>
> > >>>>>> 1) Refactor the current RPC communication for
> > >>>>>> security_groups_for_devices, which could be used for full
> > >>>>>> syncs, etc..
> > >>>>>>
> > >>>>>> 2) Benchmark && make use of a fanout queue per security
> > >>>>>> group to make sure only the hosts with instances on a
> > >>>>>> certain security group get the updates as they happen.
> > >>>>>>
> > >>>>>> @shihanzhang do you find it reasonable?
> > >>>>>>
> > >>>>>>
> > >>>>>>
> > >>>>>> ----- Original Message -----
> > >>>>>>> ----- Original Message -----
> > >>>>>>>> @Nachi: Yes that could a good improvement to factorize
> > >>>>>>>> the RPC
> > >>>>>>> mechanism.
> > >>>>>>>>
> > >>>>>>>> Another idea: What about creating a RPC topic per
> > >>>>>>>> security group (quid of the
> > >>>>>>> RPC topic
> > >>>>>>>> scalability) on which an agent subscribes if one of its
> > >>>>>>>> ports is
> > >>>>>>> associated
> > >>>>>>>> to the security group?
> > >>>>>>>>
> > >>>>>>>> Regards, Édouard.
> > >>>>>>>>
> > >>>>>>>>
> > >>>
> > >>>
> > >>>>>>> Hmm, Interesting,
> > >>>
> > >>>>>>> @Nachi, I'm not sure I fully understood:
> > >>>
> > >>>
> > >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
> > >>>>>>> port[SG_ID1, SG_ID2], port2 , port3
> > >>>
> > >>>
> > >>>>>>> Probably we may need to include also the SG_IP_LIST =
> > >>>>>>> [SG_IP1, SG_IP2] ...
> > >>>
> > >>>
> > >>>>>>> and let the agent do all the combination work.
> > >>>
> > >>>>>>> Something like this could make sense?
> > >>>
> > >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
> > >>>>>>> SG2:{IPs:[....],RULES:[....]} }
> > >>>
> > >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
> > >>>
> > >>>
> > >>>>>>> @Edouard, actually I like the idea of having the agent
> > >>>>>>> subscribed to security groups they have ports on... That
> > >>>>>>> would remove the need to include all the security groups
> > >>>>>>> information on every call...
> > >>>
> > >>>>>>> But would need another call to get the full information of
> > >>>>>>> a set of security groups at start/resync if we don't
> > >>>>>>> already have any.
> > >>>
> > >>>
> > >>>>>>>>
> > >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
> > >>>>>>> ayshihanzhang at 126.com >
> > >>>>>>>> wrote:
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> hi Miguel Ángel, I am very agree with you about the
> > >>>>>>>> following point:
> > >>>>>>>>> * physical implementation on the hosts (ipsets,
> > >>>>>>>>> nftables, ... )
> > >>>>>>>> --this can reduce the load of compute node.
> > >>>>>>>>> * rpc communication mechanisms.
> > >>>>>>>> -- this can reduce the load of neutron server can you
> > >>>>>>>> help me to review my BP specs?
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
> > >>>>>>> mangelajo at redhat.com >
> > >>>>>>>> wrote:
> > >>>>>>>>>
> > >>>>>>>>> Hi it's a very interesting topic, I was getting ready
> > >>>>>>>>> to raise the same concerns about our security groups
> > >>>>>>>>> implementation,
> > >>>>>>> shihanzhang
> > >>>>>>>>> thank you for starting this topic.
> > >>>>>>>>>
> > >>>>>>>>> Not only at low level where (with our default security
> > >>>>>>>>> group rules -allow all incoming from 'default' sg- the
> > >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
> > >>>>>>>>> "security_group_rules_for_devices" rpc call from
> > >>>>>>>>> ovs-agent to neutron-server grows to message sizes of
> > >>>>>>>>>> 100MB,
> > >>>>>>>>> generating serious scalability issues or
> > >>>>>>>>> timeouts/retries that totally break neutron service.
> > >>>>>>>>>
> > >>>>>>>>> (example trace of that RPC call with a few instances
> > >>>>>>>>> http://www.fpaste.org/104401/14008522/ )
> > >>>>>>>>>
> > >>>>>>>>> I believe that we also need to review the RPC calling
> > >>>>>>>>> mechanism for the OVS agent here, there are several
> > >>>>>>>>> possible approaches to
> > >>>>>>> breaking
> > >>>>>>>>> down (or/and CIDR compressing) the information we
> > >>>>>>>>> return via this
> > >>>>>>> api
> > >>>>>>>>> call.
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>> So we have to look at two things here:
> > >>>>>>>>>
> > >>>>>>>>> * physical implementation on the hosts (ipsets,
> > >>>>>>>>> nftables, ... ) * rpc communication mechanisms.
> > >>>>>>>>>
> > >>>>>>>>> Best regards, Miguel Ángel.
> > >>>>>>>>>
> > >>>>>>>>> ----- Mensaje original -----
> > >>>>>>>>>
> > >>>>>>>>>> Do you though about nftables that will replace
> > >>>>>>> {ip,ip6,arp,eb}tables?
> > >>>>>>>>>> It also based on the rule set mechanism. The issue
> > >>>>>>>>>> in that proposition, it's only stable since the
> > >>>>>>>>>> begin
> > >>>>>>> of the
> > >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
> > >>>>>>>>>> I don't list here (leverage iptables
> > >>>>>>> limitation,
> > >>>>>>>>>> efficient update rule, rule set, standardization of
> > >>>>>>>>>> netfilter commands...).
> > >>>>>>>>>
> > >>>>>>>>>> Édouard.
> > >>>>>>>>>
> > >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
> > >>>>>>>>>> henry4hly at gmail.com > wrote:
> > >>>>>>>>>
> > >>>>>>>>>>> we have done some tests, but have different
> > >>>>>>>>>>> result: the
> > >>>>>>> performance is
> > >>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
> > >>>>>>>>>>> but huge gap between enable/disable iptable hook
> > >>>>>>>>>>> on linux bridge
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
> > >>>>>>> ayshihanzhang at 126.com
> > >>>>>>>>>>>>
> > >>>>>>>>>>> wrote:
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>> Now I have not get accurate test data, but I can
> > >>>>>>>>>>>> confirm the following points?
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
> > >>>>>>>>>>>> is liner,
> > >>>>>>> iptable
> > >>>>>>>>>>>> filter it one by one, if a VM in default
> > >>>>>>>>>>>> security group and this default security group
> > >>>>>>>>>>>> have many members, but ipset chain is set, the
> > >>>>>>>>>>>> time
> > >>>>>>> ipset
> > >>>>>>>>>>>> filter one and many member is not much
> > >>>>>>>>>>>> difference.
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>> 2. when the iptable rule is very large, the
> > >>>>>>>>>>>> probability of
> > >>>>>>> failure
> > >>>>>>>>>>>> that iptable-save save the iptable rule is very
> > >>>>>>>>>>>> large.
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
> > >>>>>>>>>>>> blak111 at gmail.com
> > >>>>>>>> wrote:
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>>> This sounds like a good idea to handle some of
> > >>>>>>>>>>>>> the
> > >>>>>>> performance
> > >>>>>>>>>>>>> issues until the ovs firewall can be
> > >>>>>>>>>>>>> implemented down the the line.
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>> Do you have any performance comparisons?
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
> > >>>>>>> ayshihanzhang at 126.com >
> > >>>>>>>>>>>>> wrote:
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>>>> Hello all,
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>>>> Now in neutron, it use iptable implementing
> > >>>>>>>>>>>>>> security
> > >>>>>>> group, but
> > >>>>>>>>>>>>>> the performance of this implementation is
> > >>>>>>>>>>>>>> very poor, there
> > >>>>>>> is a bug:
> > >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>to
> > >>>>>>> reflect this
> > >>>>>>>>>>>>>> problem. In his test, w ith default security
> > >>>>>>>>>>>>>> groups(which has remote security group),
> > >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
> > >>>>>>>>>>>>>> Iptable rules
> > >>>>>>> on evry
> > >>>>>>>>>>>>>> compute node, although his patch can reduce
> > >>>>>>>>>>>>>> the processing time, but
> > >>>>>>> it don't
> > >>>>>>>>>>>>>> solve this problem fundamentally. I have
> > >>>>>>>>>>>>>> commit a BP to solve this
> > >>>>>>> problem:
> > >>>>>>>>>>>>>>
> > >>>>>>>
> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
> > >>>
> > >>>>>>>
> > >>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>>> There are other people interested in this
> > >>>>>>>>>>>>>> it?
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>>>> _______________________________________________
> > >>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>>> OpenStack-dev mailing list
> > >>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>>>>
> > >>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>>>>>>
> > >>>
> > >>>>>>>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>>> _______________________________________________
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>> OpenStack-dev mailing list
> > >>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
> > >>>>>>>>>>
> > >>>>>>>>>>>>
> > >>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>>>
> > >>>>>>>
> > >>>>
> > >>>>>>>>>>>>>
> > >>>>>>>>>>
> > >>>>>>>>>
> > >>>>>>>>>>> _______________________________________________
> > >>>>>>>>>>
> > >>>>>>>>>>> OpenStack-dev mailing list
> > >>>>>>>>>>
> > >>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
> > >>>>>>>>>>>
> > >>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>>>
> > >>>>>>>
> > >>>>
> > >>>>>>>>>
> > >>>>>>>>>> _______________________________________________
> > >>>>>>>>>> OpenStack-dev mailing list
> > >>>>>>>>>> OpenStack-dev at lists.openstack.org >>
> > >>>>>>>>>>
> > >>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>>>
> > >>>>>>>
> > >>>
> > >>>>>>>>> _______________________________________________
> > >>>>>>>>> OpenStack-dev mailing list
> > >>>>>>>>> OpenStack-dev at lists.openstack.org >
> > >>>>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>>>>>>>>
> > >>>>>>>>>
> > >>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> _______________________________________________
> > >>>>>>>> OpenStack-dev mailing list
> > >>>>>>>> OpenStack-dev at lists.openstack.org
> > >>>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>>>>>> _______________________________________________
> > >>>>>>>> OpenStack-dev mailing list
> > >>>>>>>> OpenStack-dev at lists.openstack.org
> > >>>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>>>>>>>
> > >>>>>>>>
> > >>>
> > >>>
> > >>>>>>> _______________________________________________
> > >>>>>>> OpenStack-dev mailing list
> > >>>>>>> OpenStack-dev at lists.openstack.org
> > >>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>>>>>>
> > >>>>>>
> > >>>>>>
> > >>>
> > >>> _______________________________________________
> > >>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>
> > >>>>>>
> > >>_______________________________________________
> > >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>
> > >>>
> > >>>>>
> > >>>>>
> > >>>> _______________________________________________ OpenStack-dev
> > >>>> mailing list OpenStack-dev at lists.openstack.org
> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>>
> > >>>>>>_______________________________________________
> > >>>>>>OpenStack-dev
> > >>>>
> > >>mailing list
> > >>>>>> OpenStack-dev at lists.openstack.org
> > >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>_______________________________________________
> > >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> > >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>>>
> > >>>>
> > >>>>_______________________________________________
> > >>>>OpenStack-dev
> > >>>>>
> > >>mailing list
> > >>>> OpenStack-dev at lists.openstack.org
> > >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> _______________________________________________ OpenStack-dev
> > >>> mailing list OpenStack-dev at lists.openstack.org
> > >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >>>
> > >>-----BEGIN PGP SIGNATURE-----
> > >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> > >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> > >>
> > >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
> > >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
> > >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
> > >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
> > >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
> > >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
> > >>=TxAT
> > >>-----END PGP SIGNATURE-----
> > >>
> > >>_______________________________________________
> > >>OpenStack-dev mailing list
> > >> OpenStack-dev at lists.openstack.org
> > >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > >
> > >
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> > --
> > Best wishes!
> > Baohua
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140821/fcb0b282/attachment-0001.html>