[openstack-dev] [neutron]Performance of security group
Ihar Hrachyshka
ihrachys at redhat.com
Thu Jul 3 14:08:21 UTC 2014
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Oh, so you have the enhancement implemented? Great! Any numbers that
shows how much we gain from that?
/Ihar
On 03/07/14 02:49, shihanzhang wrote:
> Hi, Miguel Angel Ajo! Yes, the ipset implementation is ready, today
> I will modify my spec, when the spec is approved, I will commit the
> codes as soon as possilbe!
>
>
>
>
>
> At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com>
> wrote:
>>
>> Nice Shihanzhang,
>>
>> Do you mean the ipset implementation is ready, or just the
>> spec?.
>>
>>
>> For the SG group refactor, I don't worry about who does it, or
>> who takes the credit, but I believe it's important we address
>> this bottleneck during Juno trying to match nova's scalability.
>>
>> Best regards, Miguel Ángel.
>>
>>
>> On 07/02/2014 02:50 PM, shihanzhang wrote:
>>> hi Miguel Ángel and Ihar Hrachyshka, I agree with you that
>>> split the work in several specs, I have finished the work (
>>> ipset optimization), you can do 'sg rpc optimization (without
>>> fanout)'. as the third part(sg rpc optimization (with fanout)),
>>> I think we need talk about it, because just using ipset to
>>> optimize security group agent codes does not bring the best
>>> results!
>>>
>>> Best regards, shihanzhang.
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com>
>>> wrote:
> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>
>> Shihazhang,
>
>> I really believe we need the RPC refactor done for this cycle,
>> and given the close deadlines we have (July 10 for spec
>> submission and July 20 for spec approval).
>
>> Don't you think it's going to be better to split the work in
>> several specs?
>
>> 1) ipset optimization (you) 2) sg rpc optimization (without
>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you
>> , me)
>
>
>> This way we increase the chances of having part of this for the
>> Juno cycle. If we go for something too complicated is going to
>> take more time for approval.
>
>
> I agree. And it not only increases chances to get at least some of
> those highly demanded performance enhancements to get into Juno,
> it's also "the right thing to do" (c). It's counterproductive to
> put multiple vaguely related enhancements in single spec. This
> would dim review focus and put us into position of getting
> 'all-or-nothing'. We can't afford that.
>
> Let's leave one spec per enhancement. @Shihazhang, what do you
> think?
>
>
>> Also, I proposed the details of "2", trying to bring awareness
>> on the topic, as I have been working with the scale lab in Red
>> Hat to find and understand those issues, I have a very good
>> knowledge of the problem and I believe I could make a very fast
>> advance on the issue at the RPC level.
>
>> Given that, I'd like to work on this specific part, whether or
>> not we split the specs, as it's something we believe critical
>> for neutron scalability and thus, *nova parity*.
>
>> I will start a separate spec for "2", later on, if you find it
>> ok, we keep them as separate ones, if you believe having just 1
>> spec (for 1 & 2) is going be safer for juno-* approval, then we
>> can incorporate my spec in yours, but then
>> "add-ipset-to-security" is not a good spec title to put all this
>> together.
>
>
>> Best regards, Miguel Ángel.
>
>
>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>
>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my
>>> spes, but I will also optimization the RPC from security group
>>> agent to neutron server. Now the modle is
>>> 'port[rule1,rule2...], port...', I will change it to 'port[sg1,
>>> sg2..]', this can reduce the size of RPC respose message from
>>> neutron server to security group agent.
>>>
>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>>> <mangelajo at redhat.com> wrote:
>>>>
>>>>
>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>>>> work into this problem, I could file an specific spec for
>>>> the security group RPC optimization, a masterplan in two
>>>> steps:
>>>>
>>>> 1) Refactor the current RPC communication for
>>>> security_groups_for_devices, which could be used for full
>>>> syncs, etc..
>>>>
>>>> 2) Benchmark && make use of a fanout queue per security
>>>> group to make sure only the hosts with instances on a
>>>> certain security group get the updates as they happen.
>>>>
>>>> @shihanzhang do you find it reasonable?
>>>>
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> ----- Original Message -----
>>>>>> @Nachi: Yes that could a good improvement to factorize
>>>>>> the RPC
>>>>> mechanism.
>>>>>>
>>>>>> Another idea: What about creating a RPC topic per
>>>>>> security group (quid of the
>>>>> RPC topic
>>>>>> scalability) on which an agent subscribes if one of its
>>>>>> ports is
>>>>> associated
>>>>>> to the security group?
>>>>>>
>>>>>> Regards, Édouard.
>>>>>>
>>>>>>
>
>
>>>>> Hmm, Interesting,
>
>>>>> @Nachi, I'm not sure I fully understood:
>
>
>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>>>>> port[SG_ID1, SG_ID2], port2 , port3
>
>
>>>>> Probably we may need to include also the SG_IP_LIST =
>>>>> [SG_IP1, SG_IP2] ...
>
>
>>>>> and let the agent do all the combination work.
>
>>>>> Something like this could make sense?
>
>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>>>>> SG2:{IPs:[....],RULES:[....]} }
>
>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>
>
>>>>> @Edouard, actually I like the idea of having the agent
>>>>> subscribed to security groups they have ports on... That
>>>>> would remove the need to include all the security groups
>>>>> information on every call...
>
>>>>> But would need another call to get the full information of
>>>>> a set of security groups at start/resync if we don't
>>>>> already have any.
>
>
>>>>>>
>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>>>>> ayshihanzhang at 126.com >
>>>>>> wrote:
>>>>>>
>>>>>>
>>>>>>
>>>>>> hi Miguel Ángel, I am very agree with you about the
>>>>>> following point:
>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>> nftables, ... )
>>>>>> --this can reduce the load of compute node.
>>>>>>> * rpc communication mechanisms.
>>>>>> -- this can reduce the load of neutron server can you
>>>>>> help me to review my BP specs?
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>>>>> mangelajo at redhat.com >
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi it's a very interesting topic, I was getting ready
>>>>>>> to raise the same concerns about our security groups
>>>>>>> implementation,
>>>>> shihanzhang
>>>>>>> thank you for starting this topic.
>>>>>>>
>>>>>>> Not only at low level where (with our default security
>>>>>>> group rules -allow all incoming from 'default' sg- the
>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>>>>>>> "security_group_rules_for_devices" rpc call from
>>>>>>> ovs-agent to neutron-server grows to message sizes of
>>>>>>>> 100MB,
>>>>>>> generating serious scalability issues or
>>>>>>> timeouts/retries that totally break neutron service.
>>>>>>>
>>>>>>> (example trace of that RPC call with a few instances
>>>>>>> http://www.fpaste.org/104401/14008522/ )
>>>>>>>
>>>>>>> I believe that we also need to review the RPC calling
>>>>>>> mechanism for the OVS agent here, there are several
>>>>>>> possible approaches to
>>>>> breaking
>>>>>>> down (or/and CIDR compressing) the information we
>>>>>>> return via this
>>>>> api
>>>>>>> call.
>>>>>>>
>>>>>>>
>>>>>>> So we have to look at two things here:
>>>>>>>
>>>>>>> * physical implementation on the hosts (ipsets,
>>>>>>> nftables, ... ) * rpc communication mechanisms.
>>>>>>>
>>>>>>> Best regards, Miguel Ángel.
>>>>>>>
>>>>>>> ----- Mensaje original -----
>>>>>>>
>>>>>>>> Do you though about nftables that will replace
>>>>> {ip,ip6,arp,eb}tables?
>>>>>>>> It also based on the rule set mechanism. The issue
>>>>>>>> in that proposition, it's only stable since the
>>>>>>>> begin
>>>>> of the
>>>>>>>> year and on Linux kernel 3.13. But there lot of pros
>>>>>>>> I don't list here (leverage iptables
>>>>> limitation,
>>>>>>>> efficient update rule, rule set, standardization of
>>>>>>>> netfilter commands...).
>>>>>>>
>>>>>>>> Édouard.
>>>>>>>
>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>>>>>>>> henry4hly at gmail.com > wrote:
>>>>>>>
>>>>>>>>> we have done some tests, but have different
>>>>>>>>> result: the
>>>>> performance is
>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>>>>>>>>> but huge gap between enable/disable iptable hook
>>>>>>>>> on linux bridge
>>>>>>>>
>>>>>>>
>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>>>>> ayshihanzhang at 126.com
>>>>>>>>>>
>>>>>>>>> wrote:
>>>>>>>>
>>>>>>>
>>>>>>>>>> Now I have not get accurate test data, but I can
>>>>>>>>>> confirm the following points?
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> 1. In compute node, the iptable's chain of a VM
>>>>>>>>>> is liner,
>>>>> iptable
>>>>>>>>>> filter it one by one, if a VM in default
>>>>>>>>>> security group and this default security group
>>>>>>>>>> have many members, but ipset chain is set, the
>>>>>>>>>> time
>>>>> ipset
>>>>>>>>>> filter one and many member is not much
>>>>>>>>>> difference.
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> 2. when the iptable rule is very large, the
>>>>>>>>>> probability of
>>>>> failure
>>>>>>>>>> that iptable-save save the iptable rule is very
>>>>>>>>>> large.
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>>>>>>>>>> blak111 at gmail.com
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>>>> This sounds like a good idea to handle some of
>>>>>>>>>>> the
>>>>> performance
>>>>>>>>>>> issues until the ovs firewall can be
>>>>>>>>>>> implemented down the the line.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>> Do you have any performance comparisons?
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>>>>> ayshihanzhang at 126.com >
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>>>>> Hello all,
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>>>>> Now in neutron, it use iptable implementing
>>>>>>>>>>>> security
>>>>> group, but
>>>>>>>>>>>> the performance of this implementation is
>>>>>>>>>>>> very poor, there
>>>>> is a bug:
>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>>>>>>>>>>>>
>>>>>>>>>>>>
to
>>>>> reflect this
>>>>>>>>>>>> problem. In his test, w ith default security
>>>>>>>>>>>> groups(which has remote security group),
>>>>>>>>>>>> beyond 250-300 VMs, there were around 6k
>>>>>>>>>>>> Iptable rules
>>>>> on evry
>>>>>>>>>>>> compute node, although his patch can reduce
>>>>>>>>>>>> the processing time, but
>>>>> it don't
>>>>>>>>>>>> solve this problem fundamentally. I have
>>>>>>>>>>>> commit a BP to solve this
>>>>> problem:
>>>>>>>>>>>>
>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>
>>>>>
>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>> There are other people interested in this
>>>>>>>>>>>> it?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>>>>> _______________________________________________
>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>>>>
>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>>>>>>>>
>>>>>>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>
>>>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>
>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>
>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org >
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>>>>>>
>>>>>>>
>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>>>>>
>>>>>>
>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>>>>>
>>>>>>
>
>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>>>>
>>>>
>>>>
>
> _______________________________________________
>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
>
>>>>
_______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>
>>>
>>>
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev
>>
mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>>
>>>
>>>
>>>>
_______________________________________________
>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>_______________________________________________
>>OpenStack-dev
>>>
mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>>
>
>
>
>
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
iQEcBAEBCgAGBQJTtWPVAAoJEC5aWaUY1u57PyMH/3Honqp3NQN5d1crkgn2y4zR
IiMlTMoeloaLx84Fv7Ya44evA+ZX1dDIfozrig+/uuWlVXql4EIl9vcGQ2T0xvoE
WXo7eu6D4ysca1Bx0AAmNi3IY0jC3QP46V3slmOWYHW2GAwRrrWHLyuOfCubPros
7zlOEC5MRZsh1KY3fj+bX1a7dmR6QdKqnya/JdP8I0bkkYxOXAivX+gFJCTGeB23
1Ss//rr781W9mynwB2rXssUQZU3ySK7PQmMEAVZUPkPAIzbtlTfq7nbV1GPzL3Di
/qEXL0cEx57fv9l8SvqYHqVpeh09dbh3h7kKKovwgNKCpiD1oMDoWgCS+PelZFc=
=TxAT
-----END PGP SIGNATURE-----
More information about the OpenStack-dev
mailing list