[openstack-dev] [neutron]Performance of security group

shihanzhang ayshihanzhang at 126.com
Thu Jul 10 09:30:11 UTC 2014



With the deployment 'nova + neutron + openvswitch', when we bulk create about 500 VM with a default security group, the CPU usage of neutron-server and openvswitch agent is very high, especially the CPU usage of openvswitch agent will be 100%, this will cause creating VMs failed.


With the method discussed in mailist:
1) ipset optimization   (https://review.openstack.org/#/c/100761/)
3) sg rpc optimization (with fanout)  (https://review.openstack.org/#/c/104522/)
I have implement  these two scheme in my deployment,  when we again bulk create about 500 VM with a default security group, the CPU usage of openvswitch agent will reduce to 10%, even lower than 10%, so I think the iprovement of these two options are very efficient.
Who can help us to review our spec?
   Best regards, 
shihanzhang







At 2014-07-03 10:08:21, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA512
>
>Oh, so you have the enhancement implemented? Great! Any numbers that
>shows how much we gain from that?
>
>/Ihar
>
>On 03/07/14 02:49, shihanzhang wrote:
>> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
>> I will modify my spec, when the spec is approved, I will commit the
>> codes as soon as possilbe!
>> 
>> 
>> 
>> 
>> 
>> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
>> wrote:
>>> 
>>> Nice Shihanzhang,
>>> 
>>> Do you mean the ipset implementation is ready, or just the
>>> spec?.
>>> 
>>> 
>>> For the SG group refactor, I don't worry about who does it, or
>>> who takes the credit, but I believe it's important we address
>>> this bottleneck during Juno trying to match nova's scalability.
>>> 
>>> Best regards, Miguel Ángel.
>>> 
>>> 
>>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>>>> split  the work in several specs, I have finished the work (
>>>> ipset optimization), you can do 'sg rpc optimization (without 
>>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>>>> I think we need talk about it, because just using ipset to
>>>> optimize security group agent codes does not bring the best
>>>> results!
>>>> 
>>>> Best regards, shihanzhang.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
>>>> wrote:
>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>> 
>>> Shihazhang,
>> 
>>> I really believe we need the RPC refactor done for this cycle,
>>> and given the close deadlines we have (July 10 for spec
>>> submission and July 20 for spec approval).
>> 
>>> Don't you think it's going to be better to split the work in 
>>> several specs?
>> 
>>> 1) ipset optimization   (you) 2) sg rpc optimization (without 
>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>>> , me)
>> 
>> 
>>> This way we increase the chances of having part of this for the 
>>> Juno cycle. If we go for something too complicated is going to
>>> take more time for approval.
>> 
>> 
>> I agree. And it not only increases chances to get at least some of 
>> those highly demanded performance enhancements to get into Juno,
>> it's also "the right thing to do" (c). It's counterproductive to
>> put multiple vaguely related enhancements in single spec. This
>> would dim review focus and put us into position of getting
>> 'all-or-nothing'. We can't afford that.
>> 
>> Let's leave one spec per enhancement. @Shihazhang, what do you
>> think?
>> 
>> 
>>> Also, I proposed the details of "2", trying to bring awareness
>>> on the topic, as I have been working with the scale lab in Red
>>> Hat to find and understand those issues, I have a very good
>>> knowledge of the problem and I believe I could make a very fast
>>> advance on the issue at the RPC level.
>> 
>>> Given that, I'd like to work on this specific part, whether or
>>> not we split the specs, as it's something we believe critical
>>> for neutron scalability and thus, *nova parity*.
>> 
>>> I will start a separate spec for "2", later on, if you find it
>>> ok, we keep them as separate ones, if you believe having just 1
>>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>>> can incorporate my spec in yours, but then
>>> "add-ipset-to-security" is not a good spec title to put all this
>>> together.
>> 
>> 
>>> Best regards, Miguel Ángel.
>> 
>> 
>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>> 
>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>>>> spes, but I will also optimization the RPC from security group
>>>> agent to neutron server. Now the modle is
>>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>>>> sg2..]', this can reduce the size of RPC respose message from
>>>> neutron server to security group agent.
>>>> 
>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo" 
>>>> <mangelajo at redhat.com> wrote:
>>>>> 
>>>>> 
>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to 
>>>>> work into this problem, I could file an specific spec for
>>>>> the security group RPC optimization, a masterplan in two
>>>>> steps:
>>>>> 
>>>>> 1) Refactor the current RPC communication for 
>>>>> security_groups_for_devices, which could be used for full 
>>>>> syncs, etc..
>>>>> 
>>>>> 2) Benchmark && make use of a fanout queue per security
>>>>> group to make sure only the hosts with instances on a
>>>>> certain security group get the updates as they happen.
>>>>> 
>>>>> @shihanzhang do you find it reasonable?
>>>>> 
>>>>> 
>>>>> 
>>>>> ----- Original Message -----
>>>>>> ----- Original Message -----
>>>>>>> @Nachi: Yes that could a good improvement to factorize
>>>>>>> the RPC
>>>>>> mechanism.
>>>>>>> 
>>>>>>> Another idea: What about creating a RPC topic per
>>>>>>> security group (quid of the
>>>>>> RPC topic
>>>>>>> scalability) on which an agent subscribes if one of its 
>>>>>>> ports is
>>>>>> associated
>>>>>>> to the security group?
>>>>>>> 
>>>>>>> Regards, Édouard.
>>>>>>> 
>>>>>>> 
>> 
>> 
>>>>>> Hmm, Interesting,
>> 
>>>>>> @Nachi, I'm not sure I fully understood:
>> 
>> 
>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] .. 
>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>> 
>> 
>>>>>> Probably we may need to include also the SG_IP_LIST = 
>>>>>> [SG_IP1, SG_IP2] ...
>> 
>> 
>>>>>> and let the agent do all the combination work.
>> 
>>>>>> Something like this could make sense?
>> 
>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....], 
>>>>>> SG2:{IPs:[....],RULES:[....]} }
>> 
>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>> 
>> 
>>>>>> @Edouard, actually I like the idea of having the agent 
>>>>>> subscribed to security groups they have ports on... That 
>>>>>> would remove the need to include all the security groups 
>>>>>> information on every call...
>> 
>>>>>> But would need another call to get the full information of
>>>>>> a set of security groups at start/resync if we don't
>>>>>> already have any.
>> 
>> 
>>>>>>> 
>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>>>>>> ayshihanzhang at 126.com >
>>>>>>> wrote:
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> hi Miguel Ángel, I am very agree with you about the 
>>>>>>> following point:
>>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>>> nftables, ... )
>>>>>>> --this can reduce the load of compute node.
>>>>>>>> * rpc communication mechanisms.
>>>>>>> -- this can reduce the load of neutron server can you
>>>>>>> help me to review my BP specs?
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> 
>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>>>>>> mangelajo at redhat.com >
>>>>>>> wrote:
>>>>>>>> 
>>>>>>>> Hi it's a very interesting topic, I was getting ready
>>>>>>>> to raise the same concerns about our security groups 
>>>>>>>> implementation,
>>>>>> shihanzhang
>>>>>>>> thank you for starting this topic.
>>>>>>>> 
>>>>>>>> Not only at low level where (with our default security 
>>>>>>>> group rules -allow all incoming from 'default' sg- the 
>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the 
>>>>>>>> "security_group_rules_for_devices" rpc call from 
>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>>>>>>>>> 100MB,
>>>>>>>> generating serious scalability issues or
>>>>>>>> timeouts/retries that totally break neutron service.
>>>>>>>> 
>>>>>>>> (example trace of that RPC call with a few instances 
>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>>>>>>>> 
>>>>>>>> I believe that we also need to review the RPC calling 
>>>>>>>> mechanism for the OVS agent here, there are several 
>>>>>>>> possible approaches to
>>>>>> breaking
>>>>>>>> down (or/and CIDR compressing) the information we
>>>>>>>> return via this
>>>>>> api
>>>>>>>> call.
>>>>>>>> 
>>>>>>>> 
>>>>>>>> So we have to look at two things here:
>>>>>>>> 
>>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>>> nftables, ... ) * rpc communication mechanisms.
>>>>>>>> 
>>>>>>>> Best regards, Miguel Ángel.
>>>>>>>> 
>>>>>>>> ----- Mensaje original -----
>>>>>>>> 
>>>>>>>>> Do you though about nftables that will replace
>>>>>> {ip,ip6,arp,eb}tables?
>>>>>>>>> It also based on the rule set mechanism. The issue
>>>>>>>>> in that proposition, it's only stable since the
>>>>>>>>> begin
>>>>>> of the
>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>>>>>>>>> I don't list here (leverage iptables
>>>>>> limitation,
>>>>>>>>> efficient update rule, rule set, standardization of 
>>>>>>>>> netfilter commands...).
>>>>>>>> 
>>>>>>>>> Édouard.
>>>>>>>> 
>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly < 
>>>>>>>>> henry4hly at gmail.com > wrote:
>>>>>>>> 
>>>>>>>>>> we have done some tests, but have different
>>>>>>>>>> result: the
>>>>>> performance is
>>>>>>>>>> nearly the same for empty and 5k rules in iptable, 
>>>>>>>>>> but huge gap between enable/disable iptable hook
>>>>>>>>>> on linux bridge
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>>>>>> ayshihanzhang at 126.com
>>>>>>>>>>> 
>>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> Now I have not get accurate test data, but I can 
>>>>>>>>>>> confirm the following points?
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>>>>>>>>>>> is liner,
>>>>>> iptable
>>>>>>>>>>> filter it one by one, if a VM in default
>>>>>>>>>>> security group and this default security group
>>>>>>>>>>> have many members, but ipset chain is set, the
>>>>>>>>>>> time
>>>>>> ipset
>>>>>>>>>>> filter one and many member is not much
>>>>>>>>>>> difference.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> 2. when the iptable rule is very large, the 
>>>>>>>>>>> probability of
>>>>>> failure
>>>>>>>>>>> that iptable-save save the iptable rule is very 
>>>>>>>>>>> large.
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" < 
>>>>>>>>>>> blak111 at gmail.com
>>>>>>> wrote:
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>>> This sounds like a good idea to handle some of 
>>>>>>>>>>>> the
>>>>>> performance
>>>>>>>>>>>> issues until the ovs firewall can be
>>>>>>>>>>>> implemented down the the line.
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>>> Do you have any performance comparisons?
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>>>>>> ayshihanzhang at 126.com >
>>>>>>>>>>>> wrote:
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>>>> Now in neutron, it use iptable implementing 
>>>>>>>>>>>>> security
>>>>>> group, but
>>>>>>>>>>>>> the performance of this implementation is
>>>>>>>>>>>>> very poor, there
>>>>>> is a bug:
>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>>>>>>>>>>>>>
>>>>>>>>>>>>> 
>to
>>>>>> reflect this
>>>>>>>>>>>>> problem. In his test, w ith default security 
>>>>>>>>>>>>> groups(which has remote security group),
>>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>>>>>>>>>>>>> Iptable rules
>>>>>> on evry
>>>>>>>>>>>>> compute node, although his patch can reduce
>>>>>>>>>>>>> the processing time, but
>>>>>> it don't
>>>>>>>>>>>>> solve this problem fundamentally. I have
>>>>>>>>>>>>> commit a BP to solve this
>>>>>> problem:
>>>>>>>>>>>>> 
>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>>
>>>>>> 
>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>>>> There are other people interested in this
>>>>>>>>>>>>> it?
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>>>> _______________________________________________
>>
>>>>>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>>>> 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>>>>> 
>> 
>>>>>>>>>>>>>> 
>>>>>>>>>>> 
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>>> _______________________________________________
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>>>>>>>>> 
>>>>>>>>>>> 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>> 
>>> 
>>>>>>>>>>>> 
>>>>>>>>> 
>>>>>>>> 
>>>>>>>>>> _______________________________________________
>>>>>>>>> 
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> 
>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>> 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>> 
>>> 
>>>>>>>> 
>>>>>>>>> _______________________________________________ 
>>>>>>>>> OpenStack-dev mailing list 
>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>> 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>> 
>> 
>>>>>>>> _______________________________________________ 
>>>>>>>> OpenStack-dev mailing list 
>>>>>>>> OpenStack-dev at lists.openstack.org > 
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>>>>>>> 
>>>>>>>> 
>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________ 
>>>>>>> OpenStack-dev mailing list 
>>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>>>>>> 
>>>>>>> 
>> 
>>>>>>> 
>>>>>>> 
>>>>>>> _______________________________________________ 
>>>>>>> OpenStack-dev mailing list 
>>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>>>>>> 
>>>>>>> 
>> 
>> 
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org 
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>>>>> 
>>>>> 
>>>>> 
>> 
>> _______________________________________________
>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>>
>>>>> 
>_______________________________________________
>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>
>>>> 
>>>> 
>>> _______________________________________________ OpenStack-dev 
>>> mailing list OpenStack-dev at lists.openstack.org 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>_______________________________________________
>>>>>OpenStack-dev
>>> 
>mailing list
>>>>> OpenStack-dev at lists.openstack.org 
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>> 
>_______________________________________________
>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>_______________________________________________
>>>OpenStack-dev
>>>> 
>mailing list
>>> OpenStack-dev at lists.openstack.org 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>> 
>> 
>> 
>> 
>> 
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>-----BEGIN PGP SIGNATURE-----
>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>
>iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
>IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
>WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
>7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
>1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
>/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
>=TxAT
>-----END PGP SIGNATURE-----
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/2253fa52/attachment.html>


More information about the OpenStack-dev mailing list