[devstack][tempest][neutron][ironic] Trouble debugging networking issues with devstack/lib/tempest
Hey all,
We've been having networking issues with our ironic CI job and I'm a bit blocked. Included is a sample run of the job failing [0]. Our job creates a single flat public network at [1]. Everything goes fine until we hit 'lib/tempest'. The job fails trying to create a network [2].
At first I thought this could be because we configure a single physnet which is used by the 'public' network, however I've tried jumping on the node, deleting the 'public' network and creating the network mentioned in lib/tempest and it still fails with:
Error while executing command: HttpException: 503, Unable to create the network. No tenant network is available for allocation.
Does anyone have any insight into what we're not configuring properly? Any help would be greatly appreciated!
Thanks! Mike Turek <mjturek on freenode>
[0] https://dal05.objectstorage.softlayer.net/v1/AUTH_3d8e6ecb-f597-448c-8ec2-16... [1] https://github.com/openstack/devstack/blob/master/stack.sh#L1348 [2] https://github.com/openstack/devstack/blob/master/lib/tempest#L256
On 5/15/19 4:49 PM, Michael Turek wrote:
Hey all,
We've been having networking issues with our ironic CI job and I'm a bit blocked. Included is a sample run of the job failing [0]. Our job creates a single flat public network at [1]. Everything goes fine until we hit 'lib/tempest'. The job fails trying to create a network [2].
At first I thought this could be because we configure a single physnet which is used by the 'public' network, however I've tried jumping on the node, deleting the 'public' network and creating the network mentioned in lib/tempest and it still fails with:
Error while executing command: HttpException: 503, Unable to create the network. No tenant network is available for allocation.
Does anyone have any insight into what we're not configuring properly? Any help would be greatly appreciated!
Just going based on your log:
OPTS=tenant_network_types=flat
But tenant flat networks are not supported. It's typically going to be vxlan or local.
Did you change a setting or did this just start breaking recently?
-Brian
On 5/15/19 5:37 PM, Brian Haley wrote:
On 5/15/19 4:49 PM, Michael Turek wrote:
Hey all,
We've been having networking issues with our ironic CI job and I'm a bit blocked. Included is a sample run of the job failing [0]. Our job creates a single flat public network at [1]. Everything goes fine until we hit 'lib/tempest'. The job fails trying to create a network [2].
At first I thought this could be because we configure a single physnet which is used by the 'public' network, however I've tried jumping on the node, deleting the 'public' network and creating the network mentioned in lib/tempest and it still fails with:
Error while executing command: HttpException: 503, Unable to create the network. No tenant network is available for allocation.
Does anyone have any insight into what we're not configuring properly? Any help would be greatly appreciated!
Just going based on your log:
OPTS=tenant_network_types=flat
But tenant flat networks are not supported. It's typically going to be vxlan or local.
Did you change a setting or did this just start breaking recently?
-Brian
Hey Brian,
Thanks! The job had embarrassingly been broken on an issue with lib/ironic for quite some time so it's hard to say when the networking issues arose. The only thing that's changed between the last successful run and now (on our side), is that we cherry pick a patch into devstack [0]. Highly doubt it's related but figured I'd mention it.
I'll investigate properly setting the tenant network types. Thank you for the lead, much appreciated!
Thanks, Mike Turek <mjturek on freenode>
participants (2)
-
Brian Haley
-
Michael Turek