Open Stack

Since upgrading to liberty, we've noticed some very dramatic lags in the 
application of security group updates.  Experience shows that it takes 
somewhere between 15 minutes and forever for changes to take effect.

For example, I just now added a source group rule to a project:

Ingress     -     TCP     1 - 65535     -     default

(In other words, allow access from anything that's also in this project 
and has the default security group.)  We sat and watched the iptables 
for 20-30 minutes and waited for the new rules to appear, but they 
didn't.  Finally I restarted nova-compute, at which point compute went 
through all the 'Ensuring static filters' steps and the iptables rules 
finally appeared.

I removed the rule to see if the restart resolved something but, nope, 
still waiting for the change to apply.

Additional details:

-  inter-service communication is otherwise working fine; For example, I 
can create new instances on this host and it talks to designate.

- The project in question has hundreds of instances.  When I experiment 
with smaller projects things are typically more responsive.

Is this a known issue, or something others have run into?

-Andrew