[Openstack-operators] Cinder API with multiple regions not working.
Joe Topjian
joe at topjian.net
Wed Dec 9 16:48:54 UTC 2015
Hi Salman,
Someone mentioned this same issue yesterday in relation to Terraform (maybe
a colleague of yours?), so given the two occurrences, I thought I'd look
into this.
I have a Liberty environment readily available, so I created a second set
of volume and volumev2 endpoints for a fictional region. Everything worked
as expected, so I started reviewing the config files and saw that
/etc/cinder/cinder.conf had an option
[DEFAULT]
os_region_name = RegionOne
I commented that out, but things still worked.
Then in /etc/nova/nova.conf, I saw:
[cinder]
os_region_name = RegionOne
commenting this out caused volume attachments to hang indefinitely because
nova was trying to contact cinder at RegionTwo (I'm assuming this is the
first catalog entry that was returned).
Given this is a Liberty environment, it's not accurately reproducing your
problem, but could you check and see if you have that option set in
nova.conf?
I have a Kilo environment in the process of building. Once it has finished,
I'll see if I can reproduce your error there.
Thanks,
Joe
On Wed, Dec 9, 2015 at 4:35 AM, Salman Toor <salman.toor at it.uu.se> wrote:
> Hi,
>
> I am using Kilo release on CentOS. We have recently enabled multiple
> regions and it seems that Cinder have some problems with multiple
> endpoints.
>
> Thinks are working fine with nova but cinder is behaving strange. Here are
> my endpoints
>
>
> ————
> *[root at controller: ~]* # openstack service list
> +----------------------------------+----------------+--------------+
> | ID | Name | Type |
> +----------------------------------+----------------+--------------+
> | 0a33e6f259794ff2a99e626be37c0c2b | cinderv2-hpc2n | volumev2 |
> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute |
> | 2c7828120c294d3f82e3a17835babb85 | neutron | network |
> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute |
> | 478eff4e96464ae8a958ba29f750b14c | glance | image |
> | 4a5a771d915e43c28e66538b8bc6e625 | cinder | volume |
> | 72d1be82b2e5478dbf0f3fb9e7ba969d | cinderv2 | volumev2 |
> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image |
> | a985795b49e2440db82970b81248c86e | cinder-hpc2n | volume |
> | dccd39b92ab547ddaf9047b38620145a | swift | object-store |
> | ebb1660d1d9746759a48de921521bfad | keystone | identity |
> +----------------------------------+----------------+--------------+
>
> *[root at controller: ~]* # openstack endpoint
> show a985795b49e2440db82970b81248c86e
> +--------------+--------------------------------------------------+
> | Field | Value |
> +--------------+--------------------------------------------------+
> | adminurl | http://<URL>:8776/v1/%(tenant_id)s |
> | enabled | True |
> | id | d4003e91ddf24cfb9fa497da81b01a18 |
> | internalurl | http://<URL>:8776/v1/%(tenant_id)s |
> | publicurl | http://<URL>:8776/v1/%(tenant_id)s |
> | region | HPC2N |
> | service_id | a985795b49e2440db82970b81248c86e |
> | service_name | cinder-hpc2n |
> | service_type | volume |
> +--------------+--------------------------------------------------+
>
> *[root at controller: ~]* # openstack endpoint
> show 4a5a771d915e43c28e66538b8bc6e625
> +--------------+------------------------------------------------+
> | Field | Value |
> +--------------+------------------------------------------------+
> | adminurl | http://<URL>:8776/v1/%(tenant_id)s |
> | enabled | True |
> | id | 5f19c0b535674dbd9e318c7b6d61b3bc |
> | internalurl | http://<URL>:8776/v1/%(tenant_id)s |
> | publicurl | http://<URL>:8776/v1/%(tenant_id)s |
> | region | regionOne |
> | service_id | 4a5a771d915e43c28e66538b8bc6e625 |
> | service_name | cinder |
> | service_type | volume |
> +--------------+------------------------------------------------+
>
> And same for v2 endpoints
>
> ————
>
> ——— nova-api.log ———
>
> achmentController object at 0x598a3d0>>, body: {"volumeAttachment":
> {"device": "", "volumeId": "93d96eab-e3fd-4131-9549-ed51e7299da2"}}
> _process_stack
> /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:780
> 2015-12-08 12:52:05.847 3376 INFO
> nova.api.openstack.compute.contrib.volumes
> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Attach volume
> 93d96eab-e3fd-4131-9549-ed51e7299da2 to instance
> 3a4c8722-52a7-48f2-beb7-db8938698a0d at
> 2015-12-08 12:52:05.847 3376 DEBUG nova.compute.api
> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] [instance:
> 3a4c8722-52a7-48f2-beb7-db8938698a0d] Fetching instance by UUID get
> /usr/lib/python2.7/site-packages/nova/compute/api.py:1911
> 2015-12-08 12:52:05.988 3376 ERROR nova.api.openstack
> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Caught error: internalURL endpoint
> for volume service named cinder not found
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack Traceback (most
> recent call last):
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line
> 125, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> req.get_response(self.application)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack application,
> catch_exc_info=False)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in
> call_application
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack app_iter =
> application(self.environ, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> resp(environ, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
> line 634, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> self._call_app(env, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
> line 554, in _call_app
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> self._app(env, _fake_start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> resp(environ, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> resp(environ, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
> __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack response =
> self.app(environ, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> resp(environ, start_response)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack resp =
> self.call_func(req, *args, **self.kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> self.func(req, *args, **kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in
> __call__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack content_type,
> body, accept)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in
> _process_stack
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack action_result =
> self.dispatch(meth, request, action_args)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in
> dispatch
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> method(req=request, **action_args)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/volumes.py",
> line 305, in create
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_id,
> device)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 230, in wrapped
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> func(self, context, target, *args, **kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 219, in inner
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> function(self, context, instance, *args, **kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 200, in inner
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return f(self,
> context, instance, *args, **kw)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3035, in
> attach_volume
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack disk_bus,
> device_type)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3016, in
> _attach_volume
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
> volume_bdm.destroy()
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
> __exit__
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
> six.reraise(self.type_, self.value, self.tb)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3009, in
> _attach_volume
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume =
> self.volume_api.get(context, volume_id)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in
> wrapper
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack res =
> method(self, ctx, volume_id, *args, **kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 286, in get
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack item =
> cinderclient(context).volumes.get(volume_id)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 115, in
> cinderclient
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack url =
> _SESSION.get_endpoint(auth, **service_parameters)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in
> get_endpoint
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> auth.get_endpoint(self, **kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
> line 219, in get_endpoint
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
> service_name=service_name)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in
> inner
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
> func(*args, **kwargs)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
> "/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line
> 254, in url_for
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack raise
> exceptions.EndpointNotFound(msg)
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack EndpointNotFound:
> internalURL endpoint for volume service named cinder not found
> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
>
> ———
>
> If I remove cinder-hpc2n and cinderv2-hpc2n from the service list,
> everything started to work fine.
>
> This is how I have created endpoints:
>
>
> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s
> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl http://<URL>:8776/v1/%\(tenant_id\)s
> --region regionOne 4a5a771d915e43c28e66538b8bc6e625
>
> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s
> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl http://<URL>:8776/v1/%\(tenant_id\)s
> --region HPC2N a985795b49e2440db82970b81248c86e
>
> using the service-id.
>
>
> Anyone experiencing this behavior? or Any suggestion how to fix it? Again
> regions are working fine with nova and glance.
>
>
> Regards..
> Salman.
>
>
> PhD, Scientific Computing
> Researcher, IT Department,
> Uppsala University.
> Senior Cloud Architect,
> SNIC.
> Cloud Application Expert,
> UPPMAX.
> Visiting Researcher,
> Helsinki Institute of Physics (HIP).
> salman.toor at it.uu.se
> http://www.it.uu.se/katalog/salto690
>
>
> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20151209/b259afc2/attachment.html>
More information about the OpenStack-operators
mailing list