Open Stack

Hi, Miguel Angel Ajo!
Yes, the ipset implementation is ready, today I will modify my spec, when the spec is approved, I will commit the codes as soon as possilbe!







At 2014-07-02 10:12:34, "Miguel Angel Ajo" <majopela at redhat.com> wrote:
>
>Nice Shihanzhang,
>
>   Do you mean the ipset implementation is ready, or just the spec?.
>
>
>   For the SG group refactor, I don't worry about who does it, or who
>takes the credit, but I believe it's important we address this 
>bottleneck during Juno trying to match nova's scalability.
>
>Best regards,
>Miguel Ángel.
>
>
>On 07/02/2014 02:50 PM, shihanzhang wrote:
>> hi Miguel Ángel and Ihar Hrachyshka,
>> I agree with you that split  the work in several specs, I have finished
>> the work ( ipset optimization), you can do 'sg rpc optimization (without
>> fanout)'.
>> as the third part(sg rpc optimization (with fanout)),  I think we need
>> talk about it, because just using ipset to optimize security group agent
>> codes does not bring the best results!
>>
>>      Best regards,
>> shihanzhang.
>>
>>
>>
>>
>>
>>
>>
>>
>> At 2014-07-02 04:43:24, "Ihar Hrachyshka" <ihrachys at redhat.com> wrote:
>>>-----BEGIN PGP SIGNED MESSAGE-----
>>>Hash: SHA512
>>>
>>>On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>>>
>>>> Shihazhang,
>>>>
>>>> I really believe we need the RPC refactor done for this cycle, and
>>>> given the close deadlines we have (July 10 for spec submission and
>>>> July 20 for spec approval).
>>>>
>>>> Don't you think it's going to be better to split the work in
>>>> several specs?
>>>>
>>>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
>>>> me)
>>>>
>>>>
>>>> This way we increase the chances of having part of this for the
>>>> Juno cycle. If we go for something too complicated is going to take
>>>> more time for approval.
>>>>
>>>
>>>I agree. And it not only increases chances to get at least some of
>>>those highly demanded performance enhancements to get into Juno, it's
>>>also "the right thing to do" (c). It's counterproductive to put
>>>multiple vaguely related enhancements in single spec. This would dim
>>>review focus and put us into position of getting 'all-or-nothing'. We
>>>can't afford that.
>>>
>>>Let's leave one spec per enhancement. @Shihazhang, what do you think?
>>>
>>>>
>>>> Also, I proposed the details of "2", trying to bring awareness on
>>>> the topic, as I have been working with the scale lab in Red Hat to
>>>> find and understand those issues, I have a very good knowledge of
>>>> the problem and I believe I could make a very fast advance on the
>>>> issue at the RPC level.
>>>>
>>>> Given that, I'd like to work on this specific part, whether or not
>>>> we split the specs, as it's something we believe critical for
>>>> neutron scalability and thus, *nova parity*.
>>>>
>>>> I will start a separate spec for "2", later on, if you find it ok,
>>>> we keep them as separate ones, if you believe having just 1 spec
>>>> (for 1 & 2) is going be safer for juno-* approval, then we can
>>>> incorporate my spec in yours, but then "add-ipset-to-security" is
>>>> not a good spec title to put all this together.
>>>>
>>>>
>>>> Best regards, Miguel Ángel.
>>>>
>>>>
>>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>>>
>>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
>>>>> but I will also optimization the RPC from security group agent to
>>>>> neutron server. Now the modle is 'port[rule1,rule2...], port...',
>>>>> I will change it to 'port[sg1, sg2..]', this can reduce the size
>>>>> of RPC respose message from neutron server to security group
>>>>> agent.
>>>>>
>>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>>>>> <mangelajo at redhat.com> wrote:
>>>>>>
>>>>>>
>>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>>>>>> work into this problem, I could file an specific spec for the
>>>>>> security group RPC optimization, a masterplan in two steps:
>>>>>>
>>>>>> 1) Refactor the current RPC communication for
>>>>>> security_groups_for_devices, which could be used for full
>>>>>> syncs, etc..
>>>>>>
>>>>>> 2) Benchmark && make use of a fanout queue per security group
>>>>>> to make sure only the hosts with instances on a certain
>>>>>> security group get the updates as they happen.
>>>>>>
>>>>>> @shihanzhang do you find it reasonable?
>>>>>>
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> ----- Original Message -----
>>>>>>>> @Nachi: Yes that could a good improvement to factorize the
>>>>>>>> RPC
>>>>>>> mechanism.
>>>>>>>>
>>>>>>>> Another idea: What about creating a RPC topic per security
>>>>>>>> group (quid of the
>>>>>>> RPC topic
>>>>>>>> scalability) on which an agent subscribes if one of its
>>>>>>>> ports is
>>>>>>> associated
>>>>>>>> to the security group?
>>>>>>>>
>>>>>>>> Regards, Édouard.
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Hmm, Interesting,
>>>>>>>
>>>>>>> @Nachi, I'm not sure I fully understood:
>>>>>>>
>>>>>>>
>>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>>>>>>>
>>>>>>>
>>>>>>> Probably we may need to include also the SG_IP_LIST =
>>>>>>> [SG_IP1, SG_IP2] ...
>>>>>>>
>>>>>>>
>>>>>>> and let the agent do all the combination work.
>>>>>>>
>>>>>>> Something like this could make sense?
>>>>>>>
>>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>>>>>>> SG2:{IPs:[....],RULES:[....]} }
>>>>>>>
>>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>>>>>>>
>>>>>>>
>>>>>>> @Edouard, actually I like the idea of having the agent
>>>>>>> subscribed to security groups they have ports on... That
>>>>>>> would remove the need to include all the security groups
>>>>>>> information on every call...
>>>>>>>
>>>>>>> But would need another call to get the full information of a
>>>>>>> set of security groups at start/resync if we don't already
>>>>>>> have any.
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>>>>>>> ayshihanzhang at 126.com >
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> hi Miguel Ángel, I am very agree with you about the
>>>>>>>> following point:
>>>>>>>>> * physical implementation on the hosts (ipsets, nftables,
>>>>>>>>> ... )
>>>>>>>> --this can reduce the load of compute node.
>>>>>>>>> * rpc communication mechanisms.
>>>>>>>> -- this can reduce the load of neutron server can you help
>>>>>>>> me to review my BP specs?
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>>>>>>> mangelajo at redhat.com >
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi it's a very interesting topic, I was getting ready to
>>>>>>>>> raise the same concerns about our security groups
>>>>>>>>> implementation,
>>>>>>> shihanzhang
>>>>>>>>> thank you for starting this topic.
>>>>>>>>>
>>>>>>>>> Not only at low level where (with our default security
>>>>>>>>> group rules -allow all incoming from 'default' sg- the
>>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>>>>>>>>> "security_group_rules_for_devices" rpc call from
>>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>>>>>>>>>> 100MB,
>>>>>>>>> generating serious scalability issues or timeouts/retries
>>>>>>>>> that totally break neutron service.
>>>>>>>>>
>>>>>>>>> (example trace of that RPC call with a few instances
>>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>>>>>>>>>
>>>>>>>>> I believe that we also need to review the RPC calling
>>>>>>>>> mechanism for the OVS agent here, there are several
>>>>>>>>> possible approaches to
>>>>>>> breaking
>>>>>>>>> down (or/and CIDR compressing) the information we return
>>>>>>>>> via this
>>>>>>> api
>>>>>>>>> call.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> So we have to look at two things here:
>>>>>>>>>
>>>>>>>>> * physical implementation on the hosts (ipsets, nftables,
>>>>>>>>> ... ) * rpc communication mechanisms.
>>>>>>>>>
>>>>>>>>> Best regards, Miguel Ángel.
>>>>>>>>>
>>>>>>>>> ----- Mensaje original -----
>>>>>>>>>
>>>>>>>>>> Do you though about nftables that will replace
>>>>>>> {ip,ip6,arp,eb}tables?
>>>>>>>>>> It also based on the rule set mechanism. The issue in
>>>>>>>>>> that proposition, it's only stable since the begin
>>>>>>> of the
>>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros I
>>>>>>>>>> don't list here (leverage iptables
>>>>>>> limitation,
>>>>>>>>>> efficient update rule, rule set, standardization of
>>>>>>>>>> netfilter commands...).
>>>>>>>>>
>>>>>>>>>> Édouard.
>>>>>>>>>
>>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>>>>>>>>>> henry4hly at gmail.com > wrote:
>>>>>>>>>
>>>>>>>>>>> we have done some tests, but have different result:
>>>>>>>>>>> the
>>>>>>> performance is
>>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>>>>>>>>>>> but huge gap between enable/disable iptable hook on
>>>>>>>>>>> linux bridge
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>>>>>>> ayshihanzhang at 126.com
>>>>>>>>>>>>
>>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> Now I have not get accurate test data, but I can
>>>>>>>>>>>> confirm the following points：
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM is
>>>>>>>>>>>> liner,
>>>>>>> iptable
>>>>>>>>>>>> filter it one by one, if a VM in default security
>>>>>>>>>>>> group and this default security group have many
>>>>>>>>>>>> members, but ipset chain is set, the time
>>>>>>> ipset
>>>>>>>>>>>> filter one and many member is not much difference.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> 2. when the iptable rule is very large, the
>>>>>>>>>>>> probability of
>>>>>>> failure
>>>>>>>>>>>> that iptable-save save the iptable rule is very
>>>>>>>>>>>> large.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>>>>>>>>>>>> blak111 at gmail.com
>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> This sounds like a good idea to handle some of
>>>>>>>>>>>>> the
>>>>>>> performance
>>>>>>>>>>>>> issues until the ovs firewall can be implemented
>>>>>>>>>>>>> down the the line.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> Do you have any performance comparisons?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>>>>>>> ayshihanzhang at 126.com >
>>>>>>>>>>>>> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> Now in neutron, it use iptable implementing
>>>>>>>>>>>>>> security
>>>>>>> group, but
>>>>>>>>>>>>>> the performance of this implementation is very
>>>>>>>>>>>>>> poor, there
>>>>>>> is a bug:
>>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>>>>>>>>>>>>>> to
>>>>>>> reflect this
>>>>>>>>>>>>>> problem. In his test, w ith default security
>>>>>>>>>>>>>> groups(which has remote security group), beyond
>>>>>>>>>>>>>> 250-300 VMs, there were around 6k Iptable
>>>>>>>>>>>>>> rules
>>>>>>> on evry
>>>>>>>>>>>>>> compute node, although his patch can reduce the
>>>>>>>>>>>>>> processing time, but
>>>>>>> it don't
>>>>>>>>>>>>>> solve this problem fundamentally. I have commit
>>>>>>>>>>>>>> a BP to solve this
>>>>>>> problem:
>>>>>>>>>>>>>>
>>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> There are other people interested in this it?
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>> >>
>>>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>
>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>
>>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>> >>
>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>> >
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> OpenStack-dev at lists.openstack.org >
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>>>
>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>>
>>>>
>>>>>>>
>>>>>>> _______________________________________________ OpenStack-dev
>>>>>>> mailing list OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>_______________________________________________
>>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>>
>>>_______________________________________________
>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>>
>>>> _______________________________________________ OpenStack-dev
>>>> mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>-----BEGIN PGP SIGNATURE-----
>>>Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>>Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>>
>>>iQEcBAEBCgAGBQJTs8YsAAoJEC5aWaUY1u573G8H/jfRgoJMEJ0al9+io8bKtoLK
>>>1oznScn4StQGy+eObk4cTIY1qmfEBcqdZLsGXkVtDVxtJZo/lUwViAX/r0qffazB
>>>dYCRoFqjhj+JhbQ2ul51A42vZ528lVdzwRdV0Hmwr9AdHdLwjdHi4msTqMXb0Tzs
>>>HiLIp9IED758xDN9DfCv3BTFR1EqQoczrSKmFV7eUnS7A8dTun3LzhyptD9dwzR4
>>>jKl2RJ8/rpzu+oU0R/yju6IaF6Pe48D5UOH1LjNvL3BpUBj/ULH4Z/Sw8VSV9xPj
>>>VtOowoB6evsK3bmehsiTXo/nn/M/B1Qo8n2Q+ox2A5q5/0Mhw8pH41GrOPECZlQ=
>>>=/+cP
>>>-----END PGP SIGNATURE-----
>>>
>>>_______________________________________________
>>>OpenStack-dev mailing list
>>>OpenStack-dev at lists.openstack.org
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140703/58f3790e/attachment-0001.html>