Open Stack

I have created a separate spec for the RPC part.

https://review.openstack.org/104522


On 07/02/2014 05:52 PM, Kyle Mestery wrote:
> On Wed, Jul 2, 2014 at 3:43 AM, Ihar Hrachyshka <ihrachys at redhat.com> wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> On 02/07/14 10:12, Miguel Angel Ajo wrote:
>>>
>>> Shihazhang,
>>>
>>> I really believe we need the RPC refactor done for this cycle, and
>>> given the close deadlines we have (July 10 for spec submission and
>>> July 20 for spec approval).
>>>
>>> Don't you think it's going to be better to split the work in
>>> several specs?
>>>
>>> 1) ipset optimization   (you) 2) sg rpc optimization (without
>>> fanout) (me) 3) sg rpc optimization (with fanout) (edouard, you ,
>>> me)
>>>
>>>
>>> This way we increase the chances of having part of this for the
>>> Juno cycle. If we go for something too complicated is going to take
>>> more time for approval.
>>>
>>
>> I agree. And it not only increases chances to get at least some of
>> those highly demanded performance enhancements to get into Juno, it's
>> also "the right thing to do" (c). It's counterproductive to put
>> multiple vaguely related enhancements in single spec. This would dim
>> review focus and put us into position of getting 'all-or-nothing'. We
>> can't afford that.
>>
>> Let's leave one spec per enhancement. @Shihazhang, what do you think?
>>
> +100
>
> File these as separate specs, and lets see how much of this we can get
> into Juno.
>
> Thanks for taking this enhancement and performance improvement on everyone!
>
> Kyle
>
>>>
>>> Also, I proposed the details of "2", trying to bring awareness on
>>> the topic, as I have been working with the scale lab in Red Hat to
>>> find and understand those issues, I have a very good knowledge of
>>> the problem and I believe I could make a very fast advance on the
>>> issue at the RPC level.
>>>
>>> Given that, I'd like to work on this specific part, whether or not
>>> we split the specs, as it's something we believe critical for
>>> neutron scalability and thus, *nova parity*.
>>>
>>> I will start a separate spec for "2", later on, if you find it ok,
>>> we keep them as separate ones, if you believe having just 1 spec
>>> (for 1 & 2) is going be safer for juno-* approval, then we can
>>> incorporate my spec in yours, but then "add-ipset-to-security" is
>>> not a good spec title to put all this together.
>>>
>>>
>>> Best regards, Miguel Ángel.
>>>
>>>
>>> On 07/02/2014 03:37 AM, shihanzhang wrote:
>>>>
>>>> hi Miguel Angel Ajo Pelayo! I agree with you and modify my spes,
>>>> but I will also optimization the RPC from security group agent to
>>>> neutron server. Now the modle is 'port[rule1,rule2...], port...',
>>>> I will change it to 'port[sg1, sg2..]', this can reduce the size
>>>> of RPC respose message from neutron server to security group
>>>> agent.
>>>>
>>>> At 2014-07-01 09:06:17, "Miguel Angel Ajo Pelayo"
>>>> <mangelajo at redhat.com> wrote:
>>>>>
>>>>>
>>>>> Ok, I was talking with Édouard @ IRC, and as I have time to
>>>>> work into this problem, I could file an specific spec for the
>>>>> security group RPC optimization, a masterplan in two steps:
>>>>>
>>>>> 1) Refactor the current RPC communication for
>>>>> security_groups_for_devices, which could be used for full
>>>>> syncs, etc..
>>>>>
>>>>> 2) Benchmark && make use of a fanout queue per security group
>>>>> to make sure only the hosts with instances on a certain
>>>>> security group get the updates as they happen.
>>>>>
>>>>> @shihanzhang do you find it reasonable?
>>>>>
>>>>>
>>>>>
>>>>> ----- Original Message -----
>>>>>> ----- Original Message -----
>>>>>>> @Nachi: Yes that could a good improvement to factorize the
>>>>>>> RPC
>>>>>> mechanism.
>>>>>>>
>>>>>>> Another idea: What about creating a RPC topic per security
>>>>>>> group (quid of the
>>>>>> RPC topic
>>>>>>> scalability) on which an agent subscribes if one of its
>>>>>>> ports is
>>>>>> associated
>>>>>>> to the security group?
>>>>>>>
>>>>>>> Regards, Édouard.
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>> Hmm, Interesting,
>>>>>>
>>>>>> @Nachi, I'm not sure I fully understood:
>>>>>>
>>>>>>
>>>>>> SG_LIST [ SG1, SG2] SG_RULE_LIST = [SG_Rule1, SG_Rule2] ..
>>>>>> port[SG_ID1, SG_ID2], port2 , port3
>>>>>>
>>>>>>
>>>>>> Probably we may need to include also the SG_IP_LIST =
>>>>>> [SG_IP1, SG_IP2] ...
>>>>>>
>>>>>>
>>>>>> and let the agent do all the combination work.
>>>>>>
>>>>>> Something like this could make sense?
>>>>>>
>>>>>> Security_Groups = {SG1:{IPs:[....],RULES:[....],
>>>>>> SG2:{IPs:[....],RULES:[....]} }
>>>>>>
>>>>>> Ports = {Port1:[SG1, SG2], Port2: [SG1] .... }
>>>>>>
>>>>>>
>>>>>> @Edouard, actually I like the idea of having the agent
>>>>>> subscribed to security groups they have ports on... That
>>>>>> would remove the need to include all the security groups
>>>>>> information on every call...
>>>>>>
>>>>>> But would need another call to get the full information of a
>>>>>> set of security groups at start/resync if we don't already
>>>>>> have any.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Fri, Jun 20, 2014 at 4:04 AM, shihanzhang <
>>>>>> ayshihanzhang at 126.com >
>>>>>>> wrote:
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> hi Miguel Ángel, I am very agree with you about the
>>>>>>> following point:
>>>>>>>> * physical implementation on the hosts (ipsets, nftables,
>>>>>>>> ... )
>>>>>>> --this can reduce the load of compute node.
>>>>>>>> * rpc communication mechanisms.
>>>>>>> -- this can reduce the load of neutron server can you help
>>>>>>> me to review my BP specs?
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> At 2014-06-19 10:11:34, "Miguel Angel Ajo Pelayo" <
>>>>>> mangelajo at redhat.com >
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> Hi it's a very interesting topic, I was getting ready to
>>>>>>>> raise the same concerns about our security groups
>>>>>>>> implementation,
>>>>>> shihanzhang
>>>>>>>> thank you for starting this topic.
>>>>>>>>
>>>>>>>> Not only at low level where (with our default security
>>>>>>>> group rules -allow all incoming from 'default' sg- the
>>>>>>>> iptable rules will grow in ~X^2 for a tenant, and, the
>>>>>>>> "security_group_rules_for_devices" rpc call from
>>>>>>>> ovs-agent to neutron-server grows to message sizes of
>>>>>>>>> 100MB,
>>>>>>>> generating serious scalability issues or timeouts/retries
>>>>>>>> that totally break neutron service.
>>>>>>>>
>>>>>>>> (example trace of that RPC call with a few instances
>>>>>>>> http://www.fpaste.org/104401/14008522/ )
>>>>>>>>
>>>>>>>> I believe that we also need to review the RPC calling
>>>>>>>> mechanism for the OVS agent here, there are several
>>>>>>>> possible approaches to
>>>>>> breaking
>>>>>>>> down (or/and CIDR compressing) the information we return
>>>>>>>> via this
>>>>>> api
>>>>>>>> call.
>>>>>>>>
>>>>>>>>
>>>>>>>> So we have to look at two things here:
>>>>>>>>
>>>>>>>> * physical implementation on the hosts (ipsets, nftables,
>>>>>>>> ... ) * rpc communication mechanisms.
>>>>>>>>
>>>>>>>> Best regards, Miguel Ángel.
>>>>>>>>
>>>>>>>> ----- Mensaje original -----
>>>>>>>>
>>>>>>>>> Do you though about nftables that will replace
>>>>>> {ip,ip6,arp,eb}tables?
>>>>>>>>> It also based on the rule set mechanism. The issue in
>>>>>>>>> that proposition, it's only stable since the begin
>>>>>> of the
>>>>>>>>> year and on Linux kernel 3.13. But there lot of pros I
>>>>>>>>> don't list here (leverage iptables
>>>>>> limitation,
>>>>>>>>> efficient update rule, rule set, standardization of
>>>>>>>>> netfilter commands...).
>>>>>>>>
>>>>>>>>> Édouard.
>>>>>>>>
>>>>>>>>> On Thu, Jun 19, 2014 at 8:25 AM, henry hly <
>>>>>>>>> henry4hly at gmail.com > wrote:
>>>>>>>>
>>>>>>>>>> we have done some tests, but have different result:
>>>>>>>>>> the
>>>>>> performance is
>>>>>>>>>> nearly the same for empty and 5k rules in iptable,
>>>>>>>>>> but huge gap between enable/disable iptable hook on
>>>>>>>>>> linux bridge
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> On Thu, Jun 19, 2014 at 11:21 AM, shihanzhang <
>>>>>> ayshihanzhang at 126.com
>>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>> Now I have not get accurate test data, but I can
>>>>>>>>>>> confirm the following points：
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> 1. In compute node, the iptable's chain of a VM is
>>>>>>>>>>> liner,
>>>>>> iptable
>>>>>>>>>>> filter it one by one, if a VM in default security
>>>>>>>>>>> group and this default security group have many
>>>>>>>>>>> members, but ipset chain is set, the time
>>>>>> ipset
>>>>>>>>>>> filter one and many member is not much difference.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> 2. when the iptable rule is very large, the
>>>>>>>>>>> probability of
>>>>>> failure
>>>>>>>>>>> that iptable-save save the iptable rule is very
>>>>>>>>>>> large.
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>> At 2014-06-19 10:55:56, "Kevin Benton" <
>>>>>>>>>>> blak111 at gmail.com
>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>> This sounds like a good idea to handle some of
>>>>>>>>>>>> the
>>>>>> performance
>>>>>>>>>>>> issues until the ovs firewall can be implemented
>>>>>>>>>>>> down the the line.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> Do you have any performance comparisons?
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>> On Jun 18, 2014 7:46 PM, "shihanzhang" <
>>>>>> ayshihanzhang at 126.com >
>>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>>> Hello all,
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>>> Now in neutron, it use iptable implementing
>>>>>>>>>>>>> security
>>>>>> group, but
>>>>>>>>>>>>> the performance of this implementation is very
>>>>>>>>>>>>> poor, there
>>>>>> is a bug:
>>>>>>>>>>>>> https://bugs.launchpad.net/neutron/+bug/1302272
>>>>>>>>>>>>> to
>>>>>> reflect this
>>>>>>>>>>>>> problem. In his test, w ith default security
>>>>>>>>>>>>> groups(which has remote security group), beyond
>>>>>>>>>>>>> 250-300 VMs, there were around 6k Iptable
>>>>>>>>>>>>> rules
>>>>>> on evry
>>>>>>>>>>>>> compute node, although his patch can reduce the
>>>>>>>>>>>>> processing time, but
>>>>>> it don't
>>>>>>>>>>>>> solve this problem fundamentally. I have commit
>>>>>>>>>>>>> a BP to solve this
>>>>>> problem:
>>>>>>>>>>>>>
>>>>>> https://blueprints.launchpad.net/neutron/+spec/add-ipset-to-security
>>>>>>
>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> There are other people interested in this it?
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>
>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> > > >
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>>> OpenStack-dev at lists.openstack.org >> >
>>>>>>>>>
>>>>>>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>
>>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> OpenStack-dev at lists.openstack.org >>
>>>>>>>>>
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org >
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>>>
>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>>
>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>>
>>>
>>>>>>
>>>>>> _______________________________________________ OpenStack-dev
>>>>>> mailing list OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>>
>>>>>>
>> _______________________________________________
>>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>>
>> _______________________________________________
>>>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>>
>>> _______________________________________________ OpenStack-dev
>>> mailing list OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>> Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/
>>
>> iQEcBAEBCgAGBQJTs8YsAAoJEC5aWaUY1u573G8H/jfRgoJMEJ0al9+io8bKtoLK
>> 1oznScn4StQGy+eObk4cTIY1qmfEBcqdZLsGXkVtDVxtJZo/lUwViAX/r0qffazB
>> dYCRoFqjhj+JhbQ2ul51A42vZ528lVdzwRdV0Hmwr9AdHdLwjdHi4msTqMXb0Tzs
>> HiLIp9IED758xDN9DfCv3BTFR1EqQoczrSKmFV7eUnS7A8dTun3LzhyptD9dwzR4
>> jKl2RJ8/rpzu+oU0R/yju6IaF6Pe48D5UOH1LjNvL3BpUBj/ULH4Z/Sw8VSV9xPj
>> VtOowoB6evsK3bmehsiTXo/nn/M/B1Qo8n2Q+ox2A5q5/0Mhw8pH41GrOPECZlQ=
>> =/+cP
>> -----END PGP SIGNATURE-----
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>