Open Stack

I couldn't resist making a little benchmark test of the new RPC implementation
shihanzhang wrote:

http://www.ajo.es/post/95269040924/neutron-security-group-rules-for-devices-rpc-rewrite

The results are awesome :-)

We yet need to polish the tests a bit, and it's ready.

Best regards,
Miguel Ángel.

----- Original Message -----
> On Thu, Jul 10, 2014 at 4:30 AM, shihanzhang <ayshihanzhang at 126.com> wrote:
> >
> > With the deployment 'nova + neutron + openvswitch', when we bulk create
> > about 500 VM with a default security group, the CPU usage of neutron-server
> > and openvswitch agent is very high, especially the CPU usage of openvswitch
> > agent will be 100%, this will cause creating VMs failed.
> >
> > With the method discussed in mailist:
> >
> > 1) ipset optimization   (https://review.openstack.org/#/c/100761/)
> >
> > 3) sg rpc optimization (with fanout)
> > (https://review.openstack.org/#/c/104522/)
> >
> > I have implement  these two scheme in my deployment,  when we again bulk
> > create about 500 VM with a default security group, the CPU usage of
> > openvswitch agent will reduce to 10%, even lower than 10%, so I think the
> > iprovement of these two options are very efficient.
> >
> > Who can help us to review our spec?
> >
> This is great work! These are on my list of things to review in detail
> soon, but given the Neutron sprint this week, I haven't had time yet.
> I'll try to remedy that by the weekend.
> 
> Thanks!
> Kyle
> 
> >    Best regards,
> > shihanzhang
> >
> >
> >
> >
> >
> > At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
> >>-----BEGIN PGP SIGNED MESSAGE-----
> >>Hash: SHA512
> >>
> >>Oh, so you have the enhancement implemented? Great! Any numbers that
> >>shows how much we gain from that?
> >>
> >>/Ihar
> >>
> >>On 03/07/14 02:49, shihanzhang wrote:
> >>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> >>> I will modify my spec, when the spec is approved, I will commit the
> >>> codes as soon as possilbe!
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
> >>> wrote:
> >>>>
> >>>> Nice Shihanzhang,
> >>>>
> >>>> Do you mean the ipset implementation is ready, or just the
> >>>> spec?.
> >>>>
> >>>>
> >>>> For the SG group refactor, I don't worry about who does it, or
> >>>> who takes the credit, but I believe it's important we address
> >>>> this bottleneck during Juno trying to match nova's scalability.
> >>>>
> >>>> Best regards, Miguel Ángel.
> >>>>
> >>>>
> >>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
> >>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> >>>>> split  the work in several specs, I have finished the work (
> >>>>> ipset optimization), you can do 'sg rpc optimization (without
> >>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
> >>>>> I think we need talk about it, because just using ipset to
> >>>>> optimize security group agent codes does not bring the best
> >>>>> results!
> >>>>>
> >>>>> Best regards, shihanzhang.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
> >>>>> wrote:
> >>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> >>>
> >>>> Shihazhang,
> >>>
> >>>> I really believe we need the RPC refactor done for this cycle,
> >>>> and given the close deadlines we have (July 10 for spec
> >>>> submission and July 20 for spec approval).
> >>>
> >>>> Don't you think it's going to be better to split the work in
> >>>> several specs?
> >>>
> >>>> 1) ipset optimization   (you) 2) sg rpc optimization (without
> >>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
> >>>> , me)
> >>>
> >>>
> >>>> This way we increase the chances of having part of this for the
> >>>> Juno cycle. If we go for something too complicated is going to
> >>>> take more time for approval.
> >>>
> >>>
> >>> I agree. And it not only increases chances to get at least some of
> >>> those highly demanded performance enhancements to get into Juno,
> >>> it's also "the right thing to do" (c). It's counterproductive to
> >>> put multiple vaguely related enhancements in single spec. This
> >>> would dim review focus and put us into position of getting
> >>> 'all-or-nothing'. We can't afford that.
> >>>
> >>> Let's leave one spec per enhancement. @Shihazhang, what do you
> >>> think?
> >>>
> >>>
> >>>> Also, I proposed the details of "2", trying to bring awareness
> >>>> on the topic, as I have been working with the scale lab in Red
> >>>> Hat to find and understand those issues, I have a very good
> >>>> knowledge of the problem and I believe I could make a very fast
> >>>> advance on the issue at the RPC level.
> >>>
> >>>> Given that, I'd like to work on this specific part, whether or
> >>>> not we split the specs, as it's something we believe critical
> >>>> for neutron scalability and thus, *nova parity*.
> >>>
> >>>> I will start a separate spec for "2", later on, if you find it
> >>>> ok, we keep them as separate ones, if you believe having just 1
> >>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
> >>>> can incorporate my spec in yours, but then
> >>>> "add-ipset-to-security" is not a good spec title to put all this
> >>>> together.
> >>>
> >>>
> >>>> Best regards, Miguel Ángel.
> >>>
> >>>
> >>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
> >>>>>
> >>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
> >>>>> spes, but I will also optimization the RPC from security group
> >>>>> agent to neutron server. Now the modle is
> >>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
> >>>>> sg2..]', this can reduce the size of RPC respose message from
> >>>>> neutron server to security group agent.
> >>>>>
> >>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
> >>>>> <mangelajo at redhat.com> wrote:
> >>>>>>
> >>>>>>
> >>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
> >>>>>> work into this problem, I could file an specific spec for
> >>>>>> the security group RPC optimization, a masterplan in two
> >>>>>> steps:
> >>>>>>
> >>>>>> 1) Refactor the current RPC communication for
> >>>>>> security_groups_for_devices, which could be used for full
> >>>>>> syncs, etc..
> >>>>>>
> >>>>>> 2) Benchmark && make use of a fanout queue per security
> >>>>>> group to make sure only the hosts with instances on a
> >>>>>> certain security group get the updates as they happen.
> >>>>>>
> >>>>>> @shihanzhang do you find it reasonable?
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> ----- Original Message -----
> >>>>>>> ----- Original Message -----
> >>>>>>>> @Nachi: Yes that could a good improvement to factorize
> >>>>>>>> the RPC
> >>>>>>> mechanism.
> >>>>>>>>
> >>>>>>>> Another idea: What about creating a RPC topic per
> >>>>>>>> security group (quid of the
> >>>>>>> RPC topic
> >>>>>>>> scalability) on which an agent subscribes if one of its
> >>>>>>>> ports is
> >>>>>>> associated
> >>>>>>>> to the security group?
> >>>>>>>>
> >>>>>>>> Regards, Édouard.
> >>>>>>>>
> >>>>>>>>
> >>>
> >>>
> >>>>>>> Hmm, Interesting,
> >>>
> >>>>>>> @Nachi, I'm not sure I fully understood:
> >>>
> >>>
> >>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
> >>>>>>> port[SG_ID1, SG_ID2], port2 , port3
> >>>
> >>>
> >>>>>>> Probably we may need to include also the SG_IP_LIST =
> >>>>>>> [SG_IP1, SG_IP2] ...
> >>>
> >>>
> >>>>>>> and let the agent do all the combination work.
> >>>
> >>>>>>> Something like this could make sense?
> >>>
> >>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
> >>>>>>> SG2:{IPs:[....],RULES:[....]} }
> >>>
> >>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
> >>>
> >>>
> >>>>>>> @Edouard, actually I like the idea of having the agent
> >>>>>>> subscribed to security groups they have ports on... That
> >>>>>>> would remove the need to include all the security groups
> >>>>>>> information on every call...
> >>>
> >>>>>>> But would need another call to get the full information of
> >>>>>>> a set of security groups at start/resync if we don't
> >>>>>>> already have any.
> >>>
> >>>
> >>>>>>>>
> >>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
> >>>>>>> ayshihanzhang at 126.com >
> >>>>>>>> wrote:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> hi Miguel Ángel, I am very agree with you about the
> >>>>>>>> following point:
> >>>>>>>>> * physical implementation on the hosts (ipsets,
> >>>>>>>>> nftables, ... )
> >>>>>>>> --this can reduce the load of compute node.
> >>>>>>>>> * rpc communication mechanisms.
> >>>>>>>> -- this can reduce the load of neutron server can you
> >>>>>>>> help me to review my BP specs?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
> >>>>>>> mangelajo at redhat.com >
> >>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>> Hi it's a very interesting topic, I was getting ready
> >>>>>>>>> to raise the same concerns about our security groups
> >>>>>>>>> implementation,
> >>>>>>> shihanzhang
> >>>>>>>>> thank you for starting this topic.
> >>>>>>>>>
> >>>>>>>>> Not only at low level where (with our default security
> >>>>>>>>> group rules -allow all incoming from 'default' sg- the
> >>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
> >>>>>>>>> "security_group_rules_for_devices" rpc call from
> >>>>>>>>> ovs-agent to neutron-server grows to message sizes of
> >>>>>>>>>> 100MB,
> >>>>>>>>> generating serious scalability issues or
> >>>>>>>>> timeouts/retries that totally break neutron service.
> >>>>>>>>>
> >>>>>>>>> (example trace of that RPC call with a few instances
> >>>>>>>>> http://www.fpaste.org/104401/14008522/ )
> >>>>>>>>>
> >>>>>>>>> I believe that we also need to review the RPC calling
> >>>>>>>>> mechanism for the OVS agent here, there are several
> >>>>>>>>> possible approaches to
> >>>>>>> breaking
> >>>>>>>>> down (or/and CIDR compressing) the information we
> >>>>>>>>> return via this
> >>>>>>> api
> >>>>>>>>> call.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> So we have to look at two things here:
> >>>>>>>>>
> >>>>>>>>> * physical implementation on the hosts (ipsets,
> >>>>>>>>> nftables, ... ) * rpc communication mechanisms.
> >>>>>>>>>
> >>>>>>>>> Best regards, Miguel Ángel.
> >>>>>>>>>
> >>>>>>>>> ----- Mensaje original -----
> >>>>>>>>>
> >>>>>>>>>> Do you though about nftables that will replace
> >>>>>>> {ip,ip6,arp,eb}tables?
> >>>>>>>>>> It also based on the rule set mechanism. The issue
> >>>>>>>>>> in that proposition, it's only stable since the
> >>>>>>>>>> begin
> >>>>>>> of the
> >>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
> >>>>>>>>>> I don't list here (leverage iptables
> >>>>>>> limitation,
> >>>>>>>>>> efficient update rule, rule set, standardization of
> >>>>>>>>>> netfilter commands...).
> >>>>>>>>>
> >>>>>>>>>> Édouard.
> >>>>>>>>>
> >>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
> >>>>>>>>>> henry4hly at gmail.com > wrote:
> >>>>>>>>>
> >>>>>>>>>>> we have done some tests, but have different
> >>>>>>>>>>> result: the
> >>>>>>> performance is
> >>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
> >>>>>>>>>>> but huge gap between enable/disable iptable hook
> >>>>>>>>>>> on linux bridge
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
> >>>>>>> ayshihanzhang at 126.com
> >>>>>>>>>>>>
> >>>>>>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>> Now I have not get accurate test data, but I can
> >>>>>>>>>>>> confirm the following points?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
> >>>>>>>>>>>> is liner,
> >>>>>>> iptable
> >>>>>>>>>>>> filter it one by one, if a VM in default
> >>>>>>>>>>>> security group and this default security group
> >>>>>>>>>>>> have many members, but ipset chain is set, the
> >>>>>>>>>>>> time
> >>>>>>> ipset
> >>>>>>>>>>>> filter one and many member is not much
> >>>>>>>>>>>> difference.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> 2. when the iptable rule is very large, the
> >>>>>>>>>>>> probability of
> >>>>>>> failure
> >>>>>>>>>>>> that iptable-save save the iptable rule is very
> >>>>>>>>>>>> large.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
> >>>>>>>>>>>> blak111 at gmail.com
> >>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>> This sounds like a good idea to handle some of
> >>>>>>>>>>>>> the
> >>>>>>> performance
> >>>>>>>>>>>>> issues until the ovs firewall can be
> >>>>>>>>>>>>> implemented down the the line.
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>> Do you have any performance comparisons?
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
> >>>>>>> ayshihanzhang at 126.com >
> >>>>>>>>>>>>> wrote:
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>> Now in neutron, it use iptable implementing
> >>>>>>>>>>>>>> security
> >>>>>>> group, but
> >>>>>>>>>>>>>> the performance of this implementation is
> >>>>>>>>>>>>>> very poor, there
> >>>>>>> is a bug:
> >>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>to
> >>>>>>> reflect this
> >>>>>>>>>>>>>> problem. In his test, w ith default security
> >>>>>>>>>>>>>> groups(which has remote security group),
> >>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
> >>>>>>>>>>>>>> Iptable rules
> >>>>>>> on evry
> >>>>>>>>>>>>>> compute node, although his patch can reduce
> >>>>>>>>>>>>>> the processing time, but
> >>>>>>> it don't
> >>>>>>>>>>>>>> solve this problem fundamentally. I have
> >>>>>>>>>>>>>> commit a BP to solve this
> >>>>>>> problem:
> >>>>>>>>>>>>>>
> >>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
> >>>
> >>>>>>>
> >>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>> There are other people interested in this
> >>>>>>>>>>>>>> it?
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>> _______________________________________________
> >>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>>>>>
> >>>
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>> _______________________________________________
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
> >>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>
> >>>>>>>
> >>>>
> >>>>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>
> >>>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>>
> >>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
> >>>>>>>>>>>
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>
> >>>>>>>
> >>>>
> >>>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>> OpenStack-dev at lists.openstack.org >>
> >>>>>>>>>>
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>>
> >>>>>>>
> >>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>> OpenStack-dev at lists.openstack.org >
> >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>>>>>>>
> >>>>>>>>>
> >>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> OpenStack-dev mailing list
> >>>>>>>> OpenStack-dev at lists.openstack.org
> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>>>>>>
> >>>>>>>>
> >>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> _______________________________________________
> >>>>>>>> OpenStack-dev mailing list
> >>>>>>>> OpenStack-dev at lists.openstack.org
> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>>>>>>
> >>>>>>>>
> >>>
> >>>
> >>>>>>> _______________________________________________
> >>>>>>> OpenStack-dev mailing list
> >>>>>>> OpenStack-dev at lists.openstack.org
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>>>>>
> >>>>>>
> >>>>>>
> >>>
> >>> _______________________________________________
> >>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>>
> >>>>>>
> >>_______________________________________________
> >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>
> >>>>>
> >>>>>
> >>>> _______________________________________________ OpenStack-dev
> >>>> mailing list OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>_______________________________________________
> >>>>>>OpenStack-dev
> >>>>
> >>mailing list
> >>>>>> OpenStack-dev at lists.openstack.org
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>>
> >>_______________________________________________
> >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>
> >>>>_______________________________________________
> >>>>OpenStack-dev
> >>>>>
> >>mailing list
> >>>> OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> _______________________________________________ OpenStack-dev
> >>> mailing list OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>-----BEGIN PGP SIGNATURE-----
> >>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> >>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >>
> >>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
> >>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
> >>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
> >>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
> >>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
> >>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
> >>=TxAT
> >>-----END PGP SIGNATURE-----
> >>
> >>_______________________________________________
> >>OpenStack-dev mailing list
> >>OpenStack-dev at lists.openstack.org
> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>