[openstack-dev] [neutron] [stable] Tool to aid in scalability problems mitigation.

Miguel Angel Ajo Pelayo mangelajo at redhat.com
Fri Oct 24 09:56:22 UTC 2014



----- Original Message -----
> Hi Miguel,
> 
> while we'd need to hear from the stable team, I think it's not such a bad
> idea to make this tool available to users of pre-juno openstack releases.
> As far as upstream repos are concerned, I don't know if this tool violates
> the criteria for stable branches. Even if it would be a rather large change
> for stable/icehouse, it is pretty much orthogonal to the existing code, so
> it could be ok. However, please note that stable/havana has now reached its
> EOL, so there will be no more stable release for it.

Sure, I was mentioning havana as affected, but I understand it's already
under U/S EOL, D/S distributions would always be free to backport, specially
on an orthogonal change like this.

About stable/icehouse, I'd like to hear from the stable maintainers.

> 
> The orthogonal nature of this tool however also make the case for making it
> widely available on pypi. I think it should be ok to describe the
> scalability issue in the official OpenStack Icehouse docs and point out to
> this tool for mitigation.

Yes, of course, I consider that as a second option, my point here is that 
direct upstream review time would result in better quality code here, and 
could certainly spot any hidden bugs, and increase testing quality.

It also reduces packaging time all across distributions making it available
via the standard neutron repository.


Thanks for the feedback!,

> 
> Salvatore
> 
> On 23 October 2014 14:03, Miguel Angel Ajo Pelayo < mangelajo at redhat.com >
> wrote:
> 
> 
> 
> 
> Recently, we have identified clients with problems due to the
> bad scalability of security groups in Havana and Icehouse, that
> was addressed during juno here [1] [2]
> 
> This situation is identified by blinking agents (going UP/DOWN),
> high AMQP load, nigh neutron-server load, and timeout from openvswitch
> agents when trying to contact neutron-server
> "security_group_rules_for_devices".
> 
> Doing a [1] backport involves many dependent patches related
> to the general RPC refactor in neutron (which modifies all plugins),
> and subsequent ones fixing a few bugs. Sounds risky to me. [2] Introduces
> new features and it's dependent on features which aren't available on
> all systems.
> 
> To remediate this on production systems, I wrote a quick tool
> to help on reporting security groups and mitigating the problem
> by writing almost-equivalent rules [3].
> 
> We believe this tool would be better available to the wider community,
> and under better review and testing, and, since it doesn't modify any
> behavior
> or actual code in neutron, I'd like to propose it for inclusion into, at
> least,
> Icehouse stable branch where it's more relevant.
> 
> I know the usual way is to go master->Juno->Icehouse, but at this moment
> the tool is only interesting for Icehouse (and Havana), although I believe
> it could be extended to cleanup orphaned resources, or any other cleanup
> tasks, in that case it could make sense to be available for K->J->I.
> 
> As a reference, I'm leaving links to outputs from the tool [4][5]
> 
> Looking forward to get some feedback,
> Miguel Ángel.
> 
> 
> [1] https://review.openstack.org/#/c/111876/ security group rpc refactor
> [2] https://review.openstack.org/#/c/111877/ ipset support
> [3] https://github.com/mangelajo/neutrontool
> [4] http://paste.openstack.org/show/123519/
> [5] http://paste.openstack.org/show/123525/
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 



More information about the OpenStack-dev mailing list