Open Stack

On 10/23/2013 07:22 PM, Nachi Ueno wrote:
> Hi folks
> 
> This patch was the culprit, so we have reverted it.
> https://review.openstack.org/#/c/53459/1 (Note, this is revert of revert )
> 
> However even if it is reverted, the error happens
> http://logstash.openstack.org/index.html#eyJzZWFyY2giOiJcIklwQWRkcmVzc0dlbmVyYXRpb25GYWlsdXJlQ2xpZW50OiBObyBtb3JlIElQIGFkZHJlc3NlcyBhdmFpbGFibGUgb24gbmV0d29yayBcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiODY0MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzgyNTcwMTcxMjQ3fQ==
> 
> I tested tempest in local, it works with the patch and without patch.
> # As you know, this is a timing issue.. so this doesn't mean there is no issue..
> 
> I'm reading logs, and I found some exceptions.
> https://etherpad.openstack.org/p/debug1243726
> 
> It looks like issue in IPAllocationRange.
> 
> My next culprit is the delete_subnet in ML2
> https://github.com/openstack/neutron/blob/master/neutron/plugins/ml2/plugin.py#L371
> 
>> Kyle, Bob
> Do you have any thought on this?

Hi Nachi,

Are you suggesting with_lockmode('update') is needed on the queries used
to find the ports and subnets to auto-delete in ML2's delete_network()
and delete_subnet()? I had tried that, but backed it out due to the
postgres issue with joins. I could be wrong, but I also came to the
conclusion that locking for update locks the rows returned from the
select, not the whole table, and thus would not prevent new rows from
being added concurrently with the transaction.

Or do you have another theory on what is going wrong?

Also, is this the same issue we'd been hitting for a while with the
openvswitch plugin, or is it only since we switched devstack to default
to ml2?

-Bob

> 
> Best
> Nachi
> 
> 
> 2013/10/23 Terry Wilson <twilson at redhat.com>:
>>> Hi,
>>>
>>> I just noticed several number of neutron check
>>> (tempest-devstack-vm-neutron-isolated) fails with the same error:
>>> "No more IP addresses available on network".
>>> This errors suddenly starts to occur from yesterday.
>>>
>>> I checked there is any commit merge around the time when this failure
>>> started,
>>> but there is no commit around the time.
>>> I am not sure it is a temporary issue.
>>>
>>> I files a bug in neutron: https://bugs.launchpad.net/neutron/+bug/1243726
>>>
>>> It is late in my timezone. I hope someone jumps into this issue.
>>>
>>>
>>> logstash stats is here:
>>> http://logstash.openstack.org/index.html#eyJzZWFyY2giOiJcIklwQWRkcmVzc0dlbmVyYXRpb25GYWlsdXJlQ2xpZW50OiBObyBtb3JlIElQIGFkZHJlc3NlcyBhdmFpbGFibGUgb24gbmV0d29yayBcIiIsImZpZWxkcyI6W10sIm9mZnNldCI6MCwidGltZWZyYW1lIjoiNjA0ODAwIiwiZ3JhcGhtb2RlIjoiY291bnQiLCJ0aW1lIjp7InVzZXJfaW50ZXJ2YWwiOjB9LCJzdGFtcCI6MTM4MjUzNzk0OTgxMH0=
>>>
>>> Thanks,
>>> Akihiro
>>
>> Yes. This is still happening on my patch (and many others) as well.
>>
>> Terry
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev