Open Stack

Hi Kris,

    I'm adding Shi Han Zhang to the thread,

    I'm was involved in some refactors during kilo and Han Zhang in some 
extra fixes during Liberty [1] [2] [3],

    Could you get us some logs of such failures to see what was 
happening around the failure time?, as a minimum we should
post the log error traces to a bug in https://bugs.launchpad.net/neutron

     We will be glad to use such information to make the ipset more 
fault tolerant, and try to identify the cause of the
possible race conditions.


[1] https://review.openstack.org/#/c/187483/
[2] https://review.openstack.org/190991
[3] https://review.openstack.org/#/c/187433/



Kris G. Lindgren wrote:
>
> We have been using ipsets since juno.  Twice now since our kilo 
> upgrade we have had issues with ipsets blowing up on a compute node.
>
> The first time, was iptables was referencing an ipset that was either 
> no longer there or was not added, and was trying to apply the iptables 
> config every second and dumping the full iptables-resotore output into 
> the log when it failed at TRACE level.
> Second time, was that ipsets was failing to remove an element that was 
> no longer there.
>
> For #1 I solved by restarting the neutron-openvswitch-agent.  For #2 
> we just added the entry that ipsets was trying to remove.  It seems 
> like we are having some race conditions under kilo that were not 
> present under juno (or we managed to run it for 6+ months without it 
> biting us).
>
> Is anyone else seeing the same problems?  I am noticing some commits 
> reverting/re-adding around ipsets in kilo and liberty so trying to 
> confirm if I need to open a new bug on this.
> ____________________________________________
>
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators


Kris G. Lindgren wrote:
> We have been using ipsets since juno.  Twice now since our kilo upgrade we have had issues with ipsets blowing up on a compute node.
>
> The first time, was iptables was referencing an ipset that was either no longer there or was not added, and was trying to apply the iptables config every second and dumping the full iptables-resotore output into the log when it failed at TRACE level.
> Second time, was that ipsets was failing to remove an element that was no longer there.
>
> For #1 I solved by restarting the neutron-openvswitch-agent.  For #2 we just added the entry that ipsets was trying to remove.  It seems like we are having some race conditions under kilo that were not present under juno (or we managed to run it for 6+ months without it biting us).
>
> Is anyone else seeing the same problems?  I am noticing some commits reverting/re-adding around ipsets in kilo and liberty so trying to confirm if I need to open a new bug on this.
> ____________________________________________
>
> Kris Lindgren
> Senior Linux Systems Engineer
> GoDaddy, LLC.
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators