Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On 24/10/14 11:56, Miguel Angel Ajo Pelayo wrote:
> 
> 
> ----- Original Message -----
>> Hi Miguel,
>> 
>> while we'd need to hear from the stable team, I think it's not
>> such a bad idea to make this tool available to users of pre-juno
>> openstack releases.

It's a great idea actually. It's great when code emerged from real
life downstream support cases eventually flow up to upstream for all
operator's benefit (and not just those who pay huge money for
commercial service).

>> As far as upstream repos are concerned, I don't know if this tool
>> violates the criteria for stable branches. Even if it would be a
>> rather large change for stable/icehouse, it is pretty much
>> orthogonal to the existing code, so it could be ok. However,
>> please note that stable/havana has now reached its EOL, so there
>> will be no more stable release for it.
> 
> Sure, I was mentioning havana as affected, but I understand it's
> already under U/S EOL, D/S distributions would always be free to
> backport, specially on an orthogonal change like this.
> 
> About stable/icehouse, I'd like to hear from the stable
> maintainers.

I'm for inclusion of the tool in the main neutron package. Though it's
possible to publish it on pypi as a separate package, I would better
apply formal review process to it, plus reduce packaging efforts for
distributions (and myself). The tool may be later expanded for other
useful operator hooks, so I'm for inclusion of the tool in master and
backporting it back to all supported branches.

Though official stable maintainership rules state that 'New features'
are no-go for stable branch [1], I think they should not apply in this
case since the tool does not touch production code in any way and just
provides a way to heal security groups on operator demand. Also, rules
are to break them. ;) Quoting the same document, "Proposed backports
breaking any of above guidelines can be discussed as exception
requests on openstack-stable-maint list where stable-maint team will
try to reach consensus."

Operators should be more happy if we ship such a tool as part of
neutron release and not as another third-party tool from pypi of
potentially unsafe origin.

BTW I wonder whether the tool can be useful for Juno+ setups too.
Though we mostly mitigated the problem by RPC interface rework and
ipset, some operators may still hit some limitation that could be
workarounded by optimizing their rules. Also, I think the idea of
having a tool with miscellaneous operator hooks in the master tree is
quite interesting. I would recommend to still go with pushing it to
master and then backporting to stable branches. That would also help
to get more review attention from cores than stable branch requests
usually receive. ;)

[1]: https://wiki.openstack.org/wiki/StableBranch#Appropriate_Fixes

> 
>> 
>> The orthogonal nature of this tool however also make the case for
>> making it widely available on pypi. I think it should be ok to
>> describe the scalability issue in the official OpenStack Icehouse
>> docs and point out to this tool for mitigation.
> 
> Yes, of course, I consider that as a second option, my point here
> is that direct upstream review time would result in better quality
> code here, and could certainly spot any hidden bugs, and increase
> testing quality.
> 
> It also reduces packaging time all across distributions making it
> available via the standard neutron repository.
> 
> 
> Thanks for the feedback!,
> 
>> 
>> Salvatore
>> 
>> On 23 October 2014 14:03, Miguel Angel Ajo Pelayo <
>> mangelajo at redhat.com > wrote:
>> 
>> 
>> 
>> 
>> Recently, we have identified clients with problems due to the bad
>> scalability of security groups in Havana and Icehouse, that was
>> addressed during juno here [1] [2]
>> 
>> This situation is identified by blinking agents (going UP/DOWN), 
>> high AMQP load, nigh neutron-server load, and timeout from
>> openvswitch agents when trying to contact neutron-server 
>> "security_group_rules_for_devices".
>> 
>> Doing a [1] backport involves many dependent patches related to
>> the general RPC refactor in neutron (which modifies all
>> plugins), and subsequent ones fixing a few bugs. Sounds risky to
>> me. [2] Introduces new features and it's dependent on features
>> which aren't available on all systems.
>> 
>> To remediate this on production systems, I wrote a quick tool to
>> help on reporting security groups and mitigating the problem by
>> writing almost-equivalent rules [3].
>> 
>> We believe this tool would be better available to the wider
>> community, and under better review and testing, and, since it
>> doesn't modify any behavior or actual code in neutron, I'd like
>> to propose it for inclusion into, at least, Icehouse stable
>> branch where it's more relevant.
>> 
>> I know the usual way is to go master->Juno->Icehouse, but at this
>> moment the tool is only interesting for Icehouse (and Havana),
>> although I believe it could be extended to cleanup orphaned
>> resources, or any other cleanup tasks, in that case it could make
>> sense to be available for K->J->I.
>> 
>> As a reference, I'm leaving links to outputs from the tool
>> [4][5]
>> 
>> Looking forward to get some feedback, Miguel Ángel.
>> 
>> 
>> [1] https://review.openstack.org/#/c/111876/ security group rpc
>> refactor [2] https://review.openstack.org/#/c/111877/ ipset
>> support [3] https://github.com/mangelajo/neutrontool [4]
>> http://paste.openstack.org/show/123519/ [5]
>> http://paste.openstack.org/show/123525/
>> 
>> _______________________________________________ OpenStack-dev
>> mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>> 
_______________________________________________
>> OpenStack-dev mailing list OpenStack-dev at lists.openstack.org 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>> 
> _______________________________________________ OpenStack-dev
> mailing list OpenStack-dev at lists.openstack.org 
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
-----BEGIN PGP SIGNATURE-----
Version: GnuPG/MacGPG2 v2.0.22 (Darwin)

iQEcBAEBCgAGBQJUSjIHAAoJEC5aWaUY1u57dUsH/0hs0OUJQYJJ5Ll/DbT8BgnD
sQvYMVVzVQDKU58qLYT0j03+TfdPOHcPrQ9q0/Oi2kzzpAsFS9/ZY/apIX9ztvbQ
BkCQ8tubQEy8v7gpaUgxP6tREY36g0lLWpO9D2YhNXMhzlugzbgME5s6NEa3yrNM
sXQJH9F3KpsppNtBdlUNImvcNB4xOD2IY6nQMbPTBX9CXY2y7pLDLYZchVXlexuB
S5lRKuKN3E6N9uGOUGeAqfGT0ghJZyUFNNsSuNG9H4338jXQFayGHjKiJsde8tK1
IyOYMNT9sULMpDlrVjh0+u9YZyV5bKQTjM/ldbjoRM/he1scTxe2XWe1xO4lmfY=
=0r7e
-----END PGP SIGNATURE-----