[openstack-dev] [neutron]Performance of security group
Miguel Angel Ajo Pelayo
mangelajo at redhat.com
Thu Jul 10 11:06:39 UTC 2014
Wow Shihanzhang, truly awesome!,
Is the current implementation for https://review.openstack.org/#/c/104522/
also ready?, could you make it available?.
Good work!,
----- Original Message -----
>
> With the deployment 'nova + neutron + openvswitch', when we bulk create about
> 500 VM with a default security group, the CPU usage of neutron-server and
> openvswitch agent is very high, especially the CPU usage of openvswitch
> agent will be 100%, this will cause creating VMs failed.
>
> With the method discussed in mailist:
> 1) ipset optimization (https://review.openstack.org/#/c/100761/)
> 3) sg rpc optimization (with fanout) (
> https://review.openstack.org/#/c/104522/)
> I have implement these two scheme in my deployment, when we again bulk
> create about 500 VM with a default security group, the CPU usage of
> openvswitch agent will reduce to 10%, e ven lower than 10%, so I think the i
> provement of these two options are very efficient.
> Who can help us to review our spec?
> Best regards,
> shihanzhang
>
>
>
>
>
> At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
> >-----BEGIN PGP SIGNED MESSAGE-----
> >Hash: SHA512
> >
> >Oh, so you have the enhancement implemented? Great! Any numbers that
> >shows how much we gain from that?
> >
> >/Ihar
> >
> >On 03/07/14 02:49, shihanzhang wrote:
> >> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> >> I will modify my spec, when the spec is approved, I will commit the
> >> codes as soon as possilbe!
> >>
> >>
> >>
> >>
> >>
> >> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
> >> wrote:
> >>>
> >>> Nice Shihanzhang,
> >>>
> >>> Do you mean the ipset implementation is ready, or just the
> >>> spec?.
> >>>
> >>>
> >>> For the SG group refactor, I don't worry about who does it, or
> >>> who takes the credit, but I believe it's important we address
> >>> this bottleneck during Juno trying to match nova's scalability.
> >>>
> >>> Best regards, Miguel Ángel.
> >>>
> >>>
> >>> On 07/02/2014 02:50 PM, shihanzhang wrote:
> >>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
> >>>> split the work in several specs, I have finished the work (
> >>>> ipset optimization), you can do 'sg rpc optimization (without
> >>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
> >>>> I think we need talk about it, because just using ipset to
> >>>> optimize security group agent codes does not bring the best
> >>>> results!
> >>>>
> >>>> Best regards, shihanzhang.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
> >>>> wrote:
> >> On 02/07/14 10:12, Miguel Angel Ajo wrote:
> >>
> >>> Shihazhang,
> >>
> >>> I really believe we need the RPC refactor done for this cycle,
> >>> and given the close deadlines we have (July 10 for spec
> >>> submission and July 20 for spec approval).
> >>
> >>> Don't you think it's going to be better to split the work in
> >>> several specs?
> >>
> >>> 1) ipset optimization (you) 2) sg rpc optimization (without
> >>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
> >>> , me)
> >>
> >>
> >>> This way we increase the chances of having part of this for the
> >>> Juno cycle. If we go for something too complicated is going to
> >>> take more time for approval.
> >>
> >>
> >> I agree. And it not only increases chances to get at least some of
> >> those highly demanded performance enhancements to get into Juno,
> >> it's also "the right thing to do" (c). It's counterproductive to
> >> put multiple vaguely related enhancements in single spec. This
> >> would dim review focus and put us into position of getting
> >> 'all-or-nothing'. We can't afford that.
> >>
> >> Let's leave one spec per enhancement. @Shihazhang, what do you
> >> think?
> >>
> >>
> >>> Also, I proposed the details of "2", trying to bring awareness
> >>> on the topic, as I have been working with the scale lab in Red
> >>> Hat to find and understand those issues, I have a very good
> >>> knowledge of the problem and I believe I could make a very fast
> >>> advance on the issue at the RPC level.
> >>
> >>> Given that, I'd like to work on this specific part, whether or
> >>> not we split the specs, as it's something we believe critical
> >>> for neutron scalability and thus, *nova parity*.
> >>
> >>> I will start a separate spec for "2", later on, if you find it
> >>> ok, we keep them as separate ones, if you believe having just 1
> >>> spec (for 1 & 2) is going be safer for juno-* approval, then we
> >>> can incorporate my spec in yours, but then
> >>> "add-ipset-to-security" is not a good spec title to put all this
> >>> together.
> >>
> >>
> >>> Best regards, Miguel Ángel.
> >>
> >>
> >>> On 07/02/2014 03:37 AM, shihanzhang wrote:
> >>>>
> >>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
> >>>> spes, but I will also optimization the RPC from security group
> >>>> agent to neutron server. Now the modle is
> >>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
> >>>> sg2..]', this can reduce the size of RPC respose message from
> >>>> neutron server to security group agent.
> >>>>
> >>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
> >>>> <mangelajo at redhat.com> wrote:
> >>>>>
> >>>>>
> >>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
> >>>>> work into this problem, I could file an specific spec for
> >>>>> the security group RPC optimization, a masterplan in two
> >>>>> steps:
> >>>>>
> >>>>> 1) Refactor the current RPC communication for
> >>>>> security_groups_for_devices, which could be used for full
> >>>>> syncs, etc..
> >>>>>
> >>>>> 2) Benchmark && make use of a fanout queue per security
> >>>>> group to make sure only the hosts with instances on a
> >>>>> certain security group get the updates as they happen.
> >>>>>
> >>>>> @shihanzhang do you find it reasonable?
> >>>>>
> >>>>>
> >>>>>
> >>>>> ----- Original Message -----
> >>>>>> ----- Original Message -----
> >>>>>>> @Nachi: Yes that could a good improvement to factorize
> >>>>>>> the RPC
> >>>>>> mechanism.
> >>>>>>>
> >>>>>>> Another idea: What about creating a RPC topic per
> >>>>>>> security group (quid of the
> >>>>>> RPC topic
> >>>>>>> scalability) on which an agent subscribes if one of its
> >>>>>>> ports is
> >>>>>> associated
> >>>>>>> to the security group?
> >>>>>>>
> >>>>>>> Regards, Édouard.
> >>>>>>>
> >>>>>>>
> >>
> >>
> >>>>>> Hmm, Interesting,
> >>
> >>>>>> @Nachi, I'm not sure I fully understood:
> >>
> >>
> >>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
> >>>>>> port[SG_ID1, SG_ID2], port2 , port3
> >>
> >>
> >>>>>> Probably we may need to include also the SG_IP_LIST =
> >>>>>> [SG_IP1, SG_IP2] ...
> >>
> >>
> >>>>>> and let the agent do all the combination work.
> >>
> >>>>>> Something like this could make sense?
> >>
> >>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
> >>>>>> SG2:{IPs:[....],RULES:[....]} }
> >>
> >>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
> >>
> >>
> >>>>>> @Edouard, actually I like the idea of having the agent
> >>>>>> subscribed to security groups they have ports on... That
> >>>>>> would remove the need to include all the security groups
> >>>>>> information on every call...
> >>
> >>>>>> But would need another call to get the full information of
> >>>>>> a set of security groups at start/resync if we don't
> >>>>>> already have any.
> >>
> >>
> >>>>>>>
> >>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
> >>>>>> ayshihanzhang at 126.com >
> >>>>>>> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> hi Miguel Ángel, I am very agree with you about the
> >>>>>>> following point:
> >>>>>>>> * physical implementation on the hosts (ipsets,
> >>>>>>>> nftables, ... )
> >>>>>>> --this can reduce the load of compute node.
> >>>>>>>> * rpc communication mechanisms.
> >>>>>>> -- this can reduce the load of neutron server can you
> >>>>>>> help me to review my BP specs?
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
> >>>>>> mangelajo at redhat.com >
> >>>>>>> wrote:
> >>>>>>>>
> >>>>>>>> Hi it's a very interesting topic, I was getting ready
> >>>>>>>> to raise the same concerns about our security groups
> >>>>>>>> implementation,
> >>>>>> shihanzhang
> >>>>>>>> thank you for starting this topic.
> >>>>>>>>
> >>>>>>>> Not only at low level where (with our default security
> >>>>>>>> group rules -allow all incoming from 'default' sg- the
> >>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
> >>>>>>>> "security_group_rules_for_devices" rpc call from
> >>>>>>>> ovs-agent to neutron-server grows to message sizes of
> >>>>>>>>> 100MB,
> >>>>>>>> generating serious scalability issues or
> >>>>>>>> timeouts/retries that totally break neutron service.
> >>>>>>>>
> >>>>>>>> (example trace of that RPC call with a few instances
> >>>>>>>> http://www.fpaste.org/104401/14008522/ )
> >>>>>>>>
> >>>>>>>> I believe that we also need to review the RPC calling
> >>>>>>>> mechanism for the OVS agent here, there are several
> >>>>>>>> possible approaches to
> >>>>>> breaking
> >>>>>>>> down (or/and CIDR compressing) the information we
> >>>>>>>> return via this
> >>>>>> api
> >>>>>>>> call.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> So we have to look at two things here:
> >>>>>>>>
> >>>>>>>> * physical implementation on the hosts (ipsets,
> >>>>>>>> nftables, ... ) * rpc communication mechanisms.
> >>>>>>>>
> >>>>>>>> Best regards, Miguel Ángel.
> >>>>>>>>
> >>>>>>>> ----- Mensaje original -----
> >>>>>>>>
> >>>>>>>>> Do you though about nftables that will replace
> >>>>>> {ip,ip6,arp,eb}tables?
> >>>>>>>>> It also based on the rule set mechanism. The issue
> >>>>>>>>> in that proposition, it's only stable since the
> >>>>>>>>> begin
> >>>>>> of the
> >>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
> >>>>>>>>> I don't list here (leverage iptables
> >>>>>> limitation,
> >>>>>>>>> efficient update rule, rule set, standardization of
> >>>>>>>>> netfilter commands...).
> >>>>>>>>
> >>>>>>>>> Édouard.
> >>>>>>>>
> >>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
> >>>>>>>>> henry4hly at gmail.com > wrote:
> >>>>>>>>
> >>>>>>>>>> we have done some tests, but have different
> >>>>>>>>>> result: the
> >>>>>> performance is
> >>>>>>>>>> nearly the same for empty and 5k rules in iptable,
> >>>>>>>>>> but huge gap between enable/disable iptable hook
> >>>>>>>>>> on linux bridge
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
> >>>>>> ayshihanzhang at 126.com
> >>>>>>>>>>>
> >>>>>>>>>> wrote:
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>> Now I have not get accurate test data, but I can
> >>>>>>>>>>> confirm the following points?
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
> >>>>>>>>>>> is liner,
> >>>>>> iptable
> >>>>>>>>>>> filter it one by one, if a VM in default
> >>>>>>>>>>> security group and this default security group
> >>>>>>>>>>> have many members, but ipset chain is set, the
> >>>>>>>>>>> time
> >>>>>> ipset
> >>>>>>>>>>> filter one and many member is not much
> >>>>>>>>>>> difference.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>> 2. when the iptable rule is very large, the
> >>>>>>>>>>> probability of
> >>>>>> failure
> >>>>>>>>>>> that iptable-save save the iptable rule is very
> >>>>>>>>>>> large.
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
> >>>>>>>>>>> blak111 at gmail.com
> >>>>>>> wrote:
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>> This sounds like a good idea to handle some of
> >>>>>>>>>>>> the
> >>>>>> performance
> >>>>>>>>>>>> issues until the ovs firewall can be
> >>>>>>>>>>>> implemented down the the line.
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>> Do you have any performance comparisons?
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
> >>>>>> ayshihanzhang at 126.com >
> >>>>>>>>>>>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>>> Hello all,
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>>> Now in neutron, it use iptable implementing
> >>>>>>>>>>>>> security
> >>>>>> group, but
> >>>>>>>>>>>>> the performance of this implementation is
> >>>>>>>>>>>>> very poor, there
> >>>>>> is a bug:
> >>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >to
> >>>>>> reflect this
> >>>>>>>>>>>>> problem. In his test, w ith default security
> >>>>>>>>>>>>> groups(which has remote security group),
> >>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
> >>>>>>>>>>>>> Iptable rules
> >>>>>> on evry
> >>>>>>>>>>>>> compute node, although his patch can reduce
> >>>>>>>>>>>>> the processing time, but
> >>>>>> it don't
> >>>>>>>>>>>>> solve this problem fundamentally. I have
> >>>>>>>>>>>>> commit a BP to solve this
> >>>>>> problem:
> >>>>>>>>>>>>>
> >>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
> >>
> >>>>>>
> >>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>> There are other people interested in this
> >>>>>>>>>>>>> it?
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>>> _______________________________________________
> >>
> >>>>>>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>>>>
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>>>>>
> >>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>>> _______________________________________________
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
> >>>>>>>>>
> >>>>>>>>>>>
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>
> >>>
> >>>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>>>> _______________________________________________
> >>>>>>>>>
> >>>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>>
> >>>>>>>>>> OpenStack-dev at lists.openstack.org >>
> >>>>>>>>>>
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>
> >>>
> >>>>>>>>
> >>>>>>>>> _______________________________________________
> >>>>>>>>> OpenStack-dev mailing list
> >>>>>>>>> OpenStack-dev at lists.openstack.org >>
> >>>>>>>>>
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>
> >>
> >>>>>>>> _______________________________________________
> >>>>>>>> OpenStack-dev mailing list
> >>>>>>>> OpenStack-dev at lists.openstack.org >
> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>>>>>>>
> >>>>>>>>
> >>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> OpenStack-dev mailing list
> >>>>>>> OpenStack-dev at lists.openstack.org
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>>>>>>
> >>>>>>>
> >>
> >>>>>>>
> >>>>>>>
> >>>>>>> _______________________________________________
> >>>>>>> OpenStack-dev mailing list
> >>>>>>> OpenStack-dev at lists.openstack.org
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>>>>>>
> >>>>>>>
> >>
> >>
> >>>>>> _______________________________________________
> >>>>>> OpenStack-dev mailing list
> >>>>>> OpenStack-dev at lists.openstack.org
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>>>>>
> >>>>>
> >>>>>
> >>
> >> _______________________________________________
> >>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>
> >>
> >>>>>
> >_______________________________________________
> >>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>
> >>>>
> >>>>
> >>> _______________________________________________ OpenStack-dev
> >>> mailing list OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>>_______________________________________________
> >>>>>OpenStack-dev
> >>>
> >mailing list
> >>>>> OpenStack-dev at lists.openstack.org
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>
> >>>>>
> >_______________________________________________
> >>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
> >>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>
> >>>_______________________________________________
> >>>OpenStack-dev
> >>>>
> >mailing list
> >>> OpenStack-dev at lists.openstack.org
> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>>
> >>
> >>
> >>
> >>
> >> _______________________________________________ OpenStack-dev
> >> mailing list OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >-----BEGIN PGP SIGNATURE-----
> >Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
> >Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
> >
> >iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
> >IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
> >WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
> >7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
> >1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
> >/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
> >=TxAT
> >-----END PGP SIGNATURE-----
> >
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list