Open Stack

Hi Kris,

We are moving to neutron in a month and are hitting this in our pre prod environment too.

Sam


> On 2 Sep 2015, at 6:32 am, Kris G. Lindgren <klindgren at godaddy.com> wrote:
> 
> Hello,
> 
> We ran into this again today.
> 
> I created bug: https://bugs.launchpad.net/neutron/+bug/1491131 for this.  With the log files for ~10 seconds before the issue happened to the first couple ipset delete failures.
> 
> 
> 
> 
> On 8/20/15, 6:37 AM, "Miguel Angel Ajo" <mangelajo at redhat.com> wrote:
> 
>> Hi Kris,
>> 
>>   I'm adding Shi Han Zhang to the thread,
>> 
>>   I'm was involved in some refactors during kilo and Han Zhang in some 
>> extra fixes during Liberty [1] [2] [3],
>> 
>>   Could you get us some logs of such failures to see what was 
>> happening around the failure time?, as a minimum we should
>> post the log error traces to a bug in https://bugs.launchpad.net/neutron
>> 
>>    We will be glad to use such information to make the ipset more 
>> fault tolerant, and try to identify the cause of the
>> possible race conditions.
>> 
>> 
>> [1] https://review.openstack.org/#/c/187483/
>> [2] https://review.openstack.org/190991
>> [3] https://review.openstack.org/#/c/187433/
>> 
>> 
>> 
>> Kris G. Lindgren wrote:
>>> 
>>> We have been using ipsets since juno.  Twice now since our kilo 
>>> upgrade we have had issues with ipsets blowing up on a compute node.
>>> 
>>> The first time, was iptables was referencing an ipset that was either 
>>> no longer there or was not added, and was trying to apply the iptables 
>>> config every second and dumping the full iptables-resotore output into 
>>> the log when it failed at TRACE level.
>>> Second time, was that ipsets was failing to remove an element that was 
>>> no longer there.
>>> 
>>> For #1 I solved by restarting the neutron-openvswitch-agent.  For #2 
>>> we just added the entry that ipsets was trying to remove.  It seems 
>>> like we are having some race conditions under kilo that were not 
>>> present under juno (or we managed to run it for 6+ months without it 
>>> biting us).
>>> 
>>> Is anyone else seeing the same problems?  I am noticing some commits 
>>> reverting/re-adding around ipsets in kilo and liberty so trying to 
>>> confirm if I need to open a new bug on this.
>>> ____________________________________________
>>> 
>>> Kris Lindgren
>>> Senior Linux Systems Engineer
>>> GoDaddy, LLC.
>>> 
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> 
>> 
>> Kris G. Lindgren wrote:
>>> We have been using ipsets since juno.  Twice now since our kilo upgrade we have had issues with ipsets blowing up on a compute node.
>>> 
>>> The first time, was iptables was referencing an ipset that was either no longer there or was not added, and was trying to apply the iptables config every second and dumping the full iptables-resotore output into the log when it failed at TRACE level.
>>> Second time, was that ipsets was failing to remove an element that was no longer there.
>>> 
>>> For #1 I solved by restarting the neutron-openvswitch-agent.  For #2 we just added the entry that ipsets was trying to remove.  It seems like we are having some race conditions under kilo that were not present under juno (or we managed to run it for 6+ months without it biting us).
>>> 
>>> Is anyone else seeing the same problems?  I am noticing some commits reverting/re-adding around ipsets in kilo and liberty so trying to confirm if I need to open a new bug on this.
>>> ____________________________________________
>>> 
>>> Kris Lindgren
>>> Senior Linux Systems Engineer
>>> GoDaddy, LLC.
>>> 
>>> _______________________________________________
>>> OpenStack-operators mailing list
>>> OpenStack-operators at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>> 
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators