[openstack-dev] [neutron] [stable] Tool to aid in scalability problems mitigation.

Miguel Angel Ajo Pelayo mangelajo at redhat.com
Thu Oct 23 13:03:10 UTC 2014



    Recently, we have identified clients with problems due to the 
bad scalability of security groups in Havana and Icehouse, that 
was addressed during juno here [1] [2]

    This situation is identified by blinking agents (going UP/DOWN),
high AMQP load, nigh neutron-server load, and timeout from openvswitch
agents when trying to contact neutron-server "security_group_rules_for_devices".

    Doing a [1] backport involves many dependent patches related 
to the general RPC refactor in neutron (which modifies all plugins), 
and subsequent ones fixing a few bugs. Sounds risky to me. [2] Introduces 
new features and it's dependent on features which aren't available on 
all systems.

    To remediate this on production systems, I wrote a quick tool
to help on reporting security groups and mitigating the problem
by writing almost-equivalent rules [3]. 

    We believe this tool would be better available to the wider community,
and under better review and testing, and, since it doesn't modify any behavior 
or actual code in neutron, I'd like to propose it for inclusion into, at least, 
Icehouse stable branch where it's more relevant.

    I know the usual way is to go master->Juno->Icehouse, but at this moment
the tool is only interesting for Icehouse (and Havana), although I believe 
it could be extended to cleanup orphaned resources, or any other cleanup 
tasks, in that case it could make sense to be available for K->J->I.
 
    As a reference, I'm leaving links to outputs from the tool [4][5]
      
    Looking forward to get some feedback,
Miguel Ángel.
    

[1] https://review.openstack.org/#/c/111876/ security group rpc refactor
[2] https://review.openstack.org/#/c/111877/ ipset support
[3] https://github.com/mangelajo/neutrontool
[4] http://paste.openstack.org/show/123519/
[5] http://paste.openstack.org/show/123525/



More information about the OpenStack-dev mailing list