Open Stack

Hi colleagues,

Note: errors, described below, don't have a visible impact in my current 
configuration. But I draw attention on this since need to understand 
whether it's expected behaviour or something wrong with something and 
what exactly.

When creating loadbalancer infra (balancer, listener, pool and members) 
in active/standby agents configuration, either using Neutron CLI or Heat 
orchestration (below), I see the following errors in _neutron-server.log_ :

* Upon listener create (below showed a single block of related messages, 
second is very similar to this; both when using either 
"lbaas-listener-create" command or Heat orchestration) -

2017-11-29 15:05:29.440 1021 DEBUG neutron.api.v2.base 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] Request body: {u'security_group_rule': {u'direction': 
u'ingress', u'protocol': 51, u'ethertype': u'IPv4', u'port_range_max': 
None, u'security_group_id': u'4669fde0-5d4e-40be-b560-21173ac4561f', 
u'port_range_min': None}} prepare_request_body 
/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py:695
2017-11-29 15:05:29.443 1021 DEBUG neutron.db.quota.driver 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] Resources 
member,graph,subnetpool,listener,healthmonitor,l7policy have unlimited 
quota limit. It is not required to calculate headroom make_reservation 
/usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:223
2017-11-29 15:05:29.446 1021 DEBUG neutron.quota.resource 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] Usage tracker for resource:security_group_rule and 
tenant:413f0da1f66146ed801b1f6ced1cda48 is out of sync, need to count 
used quota count_used 
/usr/lib/python2.7/dist-packages/neutron/quota/resource.py:274
2017-11-29 15:05:29.451 1021 DEBUG neutron.quota.resource 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] Quota usage for security_group_rule was recalculated. 
Used quota:24. count_used 
/usr/lib/python2.7/dist-packages/neutron/quota/resource.py:293
2017-11-29 15:05:29.454 1021 DEBUG neutron.db.quota.driver 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] Attempting to reserve 1 items for resource 
security_group_rule. Total usage: 24; quota limit: 100; headroom:76 
make_reservation 
/usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:255
2017-11-29 15:05:29.613 1021 DEBUG neutron_lib.callbacks.manager 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] Notify callbacks [] for security_group_rule, 
before_create _notify_loop 
/usr/lib/python2.7/dist-packages/neutron_lib/callbacks/manager.py:167
2017-11-29 15:05:29.721 1021 INFO neutron.api.v2.resource 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] *create failed (client error): There was a conflict 
when trying to complete your request.*
2017-11-29 15:05:29.722 1021 INFO neutron.wsgi 
[req-a78d573d-6cb8-4164-afac-e37bb340640c 
2a012afc274341dc81b7ea5140662e8c 413f0da1f66146ed801b1f6ced1cda48 - 
default default] 10.0.10.10 *"POST /v2.0/security-group-rules HTTP/1.1"* 
status: 409  len: 347 time: 0.2877431*
*
* Upon creation of pool members (2 members), these errors appear only 
when creating infra using Heat (no errors when using neutron 
"lbaas-member-create" CLI command ) :

2017-11-29 12:04:30.229 1018 DEBUG neutron.api.v2.base 
[req-e3d66438-da0c-4b68-be6c-d47b3bce55b1 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Request body: {u'member': {u'subnet_id': 
u'ecb891c1-b7e5-45e0-8815-8675381d70d2', u'protocol_port': 8080, 
u'admin_state_up': True, u'weight': 1, u'address': u'10.1.1.11'}} 
prepare_request_body 
/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py:695
2017-11-29 12:04:30.232 1018 DEBUG neutron.db.quota.driver 
[req-e3d66438-da0c-4b68-be6c-d47b3bce55b1 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Resources 
member,graph,subnetpool,listener,healthmonitor,l7policy have unlimited 
quota limit. It is not required to calculate headroom make_reservation 
/usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:223
2017-11-29 12:04:30.690 1018 INFO neutron.api.v2.resource 
[req-e3d66438-da0c-4b68-be6c-d47b3bce55b1 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] *create failed (client error): There was a conflict 
when trying to complete your request.*
2017-11-29 12:04:30.691 1018 INFO neutron.wsgi 
[req-e3d66438-da0c-4b68-be6c-d47b3bce55b1 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] 10.0.10.10 *"POST 
/v2.0/lbaas/pools/e106e039-af27-4cfa-baa2-7238acd3078e/members 
HTTP/1.1"* status: 409  len: 342 time: 0.4688179*
*
2017-11-29 12:04:31.704 1018 DEBUG neutron.api.v2.base 
[req-f8f012ba-885d-4eeb-a3ec-f8a664328926 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Request body: {u'member': {u'subnet_id': 
u'ecb891c1-b7e5-45e0-8815-8675381d70d2', u'protocol_port': 8080, 
u'admin_state_up': True, u'weight': 1, u'address': u'10.1.1.11'}} 
prepare_request_body 
/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py:695
2017-11-29 12:04:31.709 1018 DEBUG neutron.db.quota.driver 
[req-f8f012ba-885d-4eeb-a3ec-f8a664328926 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Resources 
member,graph,subnetpool,listener,healthmonitor,l7policy have unlimited 
quota limit. It is not required to calculate headroom make_reservation 
/usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:223
2017-11-29 12:04:32.268 1018 INFO neutron.api.v2.resource 
[req-f8f012ba-885d-4eeb-a3ec-f8a664328926 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] *create failed (client error): There was a conflict 
when trying to complete your request.*
2017-11-29 12:04:32.269 1018 INFO neutron.wsgi 
[req-f8f012ba-885d-4eeb-a3ec-f8a664328926 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] 10.0.10.10 *"POST 
/v2.0/lbaas/pools/e106e039-af27-4cfa-baa2-7238acd3078e/members 
HTTP/1.1"* status: 409  len: 342 time: 0.5732501*
*
2017-11-29 12:04:33.283 1018 DEBUG neutron.api.v2.base 
[req-48d0074b-07dd-48cc-a7e9-096b5bccb0c4 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Request body: {u'member': {u'subnet_id': 
u'ecb891c1-b7e5-45e0-8815-8675381d70d2', u'protocol_port': 8080, 
u'admin_state_up': True, u'weight': 1, u'address': u'10.1.1.11'}} 
prepare_request_body 
/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py:695
2017-11-29 12:04:33.286 1018 DEBUG neutron.db.quota.driver 
[req-48d0074b-07dd-48cc-a7e9-096b5bccb0c4 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Resources 
member,graph,subnetpool,listener,healthmonitor,l7policy have unlimited 
quota limit. It is not required to calculate headroom make_reservation 
/usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:223
2017-11-29 12:04:33.646 1018 INFO neutron.api.v2.resource 
[req-48d0074b-07dd-48cc-a7e9-096b5bccb0c4 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] *create failed (client error): There was a conflict 
when trying to complete your request.*
2017-11-29 12:04:33.647 1018 INFO neutron.wsgi 
[req-48d0074b-07dd-48cc-a7e9-096b5bccb0c4 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] 10.0.10.10 *"POST 
/v2.0/lbaas/pools/e106e039-af27-4cfa-baa2-7238acd3078e/members 
HTTP/1.1"* status: 409  len: 342 time: 0.3708489*
*
2017-11-29 12:04:34.659 1018 DEBUG neutron.api.v2.base 
[req-ffde5c98-d157-4835-ba87-a01897a5a9b7 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Request body: {u'member': {u'subnet_id': 
u'ecb891c1-b7e5-45e0-8815-8675381d70d2', u'protocol_port': 8080, 
u'admin_state_up': True, u'weight': 1, u'address': u'10.1.1.11'}} 
prepare_request_body 
/usr/lib/python2.7/dist-packages/neutron/api/v2/base.py:695
2017-11-29 12:04:34.663 1018 DEBUG neutron.db.quota.driver 
[req-ffde5c98-d157-4835-ba87-a01897a5a9b7 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] Resources 
member,graph,subnetpool,listener,healthmonitor,l7policy have unlimited 
quota limit. It is not required to calculate headroom make_reservation 
/usr/lib/python2.7/dist-packages/neutron/db/quota/driver.py:223
2017-11-29 12:04:35.015 1018 INFO neutron.api.v2.resource 
[req-ffde5c98-d157-4835-ba87-a01897a5a9b7 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] *create failed (client error): There was a conflict 
when trying to complete your request.*
2017-11-29 12:04:35.016 1018 INFO neutron.wsgi 
[req-ffde5c98-d157-4835-ba87-a01897a5a9b7 
e6406606bd9d48aabc413468f9703cf6 c1114776e144400da17d8e060856be8c - 
default default] 10.0.10.10 *"POST 
/v2.0/lbaas/pools/e106e039-af27-4cfa-baa2-7238acd3078e/members 
HTTP/1.1"* status: 409  len: 342 time: 0.3637619*
*
Errors above have corresponding errors in _heat-engine.log_ (there are 
two members (pm1 and pm2) in pool and no errors regarding second (pm2) 
pool member):

2017-11-29 12:04:28.927 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] starting start 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:182
2017-11-29 12:04:28.927 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
2017-11-29 12:04:28.928 6286 INFO heat.engine.resource 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] 
creating PoolMember "pm1" Stack "nbt" [b8beca77-19c7-49e5-94a7-ec079d841277]
2017-11-29 12:04:30.095 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:30.096 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
*2017-11-29 12:04:30.692 6286 DEBUG neutronclient.v2_0.client 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] 
Error message: {"NeutronError": {"message": "Invalid state 
PENDING_UPDATE of loadbalancer resource 
db8ae876-b6eb-4c45-95d1-33e0ca6193de", "type": "StateInvalid", "detail": 
""}} _handle_fault_response 
/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:258*
2017-11-29 12:04:30.692 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:31.693 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
*2017-11-29 12:04:32.270 6286 DEBUG neutronclient.v2_0.client 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] 
Error message: {"NeutronError": {"message": "Invalid state 
PENDING_UPDATE of loadbalancer resource 
db8ae876-b6eb-4c45-95d1-33e0ca6193de", "type": "StateInvalid", "detail": 
""}} _handle_fault_response 
/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:258*
2017-11-29 12:04:32.271 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:33.272 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
*2017-11-29 12:04:33.648 6286 DEBUG neutronclient.v2_0.client 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] 
Error message: {"NeutronError": {"message": "Invalid state 
PENDING_UPDATE of loadbalancer resource 
db8ae876-b6eb-4c45-95d1-33e0ca6193de", "type": "StateInvalid", "detail": 
""}} _handle_fault_response 
/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:258*
2017-11-29 12:04:33.648 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:34.649 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
*2017-11-29 12:04:35.016 6286 DEBUG neutronclient.v2_0.client 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] 
Error message: {"NeutronError": {"message": "Invalid state 
PENDING_UPDATE of loadbalancer resource 
db8ae876-b6eb-4c45-95d1-33e0ca6193de", "type": "StateInvalid", "detail": 
""}} _handle_fault_response 
/usr/lib/python2.7/dist-packages/neutronclient/v2_0/client.py:258*
2017-11-29 12:04:35.017 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:36.017 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
2017-11-29 12:04:38.763 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:39.763 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
2017-11-29 12:04:39.891 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:40.892 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
2017-11-29 12:04:41.013 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] sleeping _sleep 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:155
2017-11-29 12:04:42.013 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] running step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:214
2017-11-29 12:04:42.277 6286 DEBUG heat.engine.scheduler 
[req-346cc302-ec69-4781-9afb-1dc1474c6ebc - bush - default default] Task 
create from PoolMember "pm1" Stack "nbt" 
[b8beca77-19c7-49e5-94a7-ec079d841277] complete step 
/usr/lib/python2.7/dist-packages/heat/engine/scheduler.py:220

Heat template for loadbalancer is the following:

   balancer:
     type: OS::Neutron::LBaaS::LoadBalancer
     properties:
       name: nbt-balancer
       vip_subnet: { get_resource: lan-subnet }

   listener:
     type: OS::Neutron::LBaaS::Listener
     properties:
       name: nbt-listener
       protocol: TCP
       protocol_port: { get_param: lb_port }
       loadbalancer: { get_resource: balancer }

   pool:
     type: OS::Neutron::LBaaS::Pool
     properties:
       name: nbt-pool
       protocol: TCP
       lb_algorithm: ROUND_ROBIN
       listener: { get_resource: listener }

   pm1:
     type:  OS::Neutron::LBaaS::PoolMember
     properties:
       address: { get_attr: [ n1, first_address ]}
       pool: { get_resource: pool }
       protocol_port: { get_param: pool_port }
       subnet: { get_resource: lan-subnet }

   pm2:
     type:  OS::Neutron::LBaaS::PoolMember
     properties:
       address: { get_attr: [ n2, first_address ]}
       pool: { get_resource: pool }
       protocol_port: { get_param: pool_port }
       subnet: { get_resource: lan-subnet }

and, of course, servers n1 and n2 are exist and are operational.

I will appreciate if you'll take a look at the issue and give some 
feedback on this. I can provide any related information in order to 
clarify this issue.

Thank you.

-- 
Volodymyr Litovka
   "Vision without Execution is Hallucination." -- Thomas Edison

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171129/4bcdcaf6/attachment.html>