[Openstack-operators] Cinder API with multiple regions not working.
Joe Topjian
joe at topjian.net
Fri Dec 11 22:45:53 UTC 2015
Hi Salman,
That's awesome news. Glad it's working. :)
Joe
On Fri, Dec 11, 2015 at 3:12 PM, Salman Toor <salman.toor at it.uu.se> wrote:
> Hi,
>
> It is working by setting exact names of the services.
>
>
> *[root at smog: ~]* # openstack service list
>
> +----------------------------------+--------------+--------------+
>
> | ID | Name | Type |
>
> +----------------------------------+--------------+--------------+
>
> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute |
>
> | 2c7828120c294d3f82e3a17835babb85 | neutron | network |
>
> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute |
>
> | 478eff4e96464ae8a958ba29f750b14c | glance | image |
>
> | 61d8baeb4ee74c7798a60758b2f4171f | cinderv2 | volumev2 |
>
> | 75f89962c7864507b07055fbfc98053e | cinder | volume |
>
> | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | cinder | volume |
>
> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image |
>
> | 9d2a7ef6b36c45b096e552bf73cb89ae | cinderv2 | volumev2 |
>
> | dccd39b92ab547ddaf9047b38620145a | swift | object-store |
>
> | ebb1660d1d9746759a48de921521bfad | keystone | identity |
>
> +----------------------------------+--------------+--------------+
>
> *[root at smog: ~]* # openstack endpoint list
>
>
> +----------------------------------+-----------+--------------+--------------+
>
> | ID | Region | Service Name | Service
> Type |
>
>
> +----------------------------------+-----------+--------------+--------------+
>
> | 3000cb23c6ab4ee5b68876ee08257338 | regionOne | cinderv2 | volumev2
> |
>
> | e21e935ed861484e976d1b93e0fda0f0 | regionOne | nova | compute
> |
>
> | cdfb008c47a0472ab3a93e6ee07e9ba4 | regionOne | neutron | network
> |
>
> | e2693bcaf3da4be2810f04acd7995d7f | regionOne | cinder | volume
> |
>
> | 1400c42cbf154f63b0c3e8d64352d1f2 | HPC2N | cinderv2 | volumev2
> |
>
> | 04acd1666704433d991cf9a75957c815 | HPC2N | glance-hpc2n | image
> |
>
> | cfd1c766c56744309767cd84034f9bfb | regionOne | swift |
> object-store |
>
> | abe46f0d86064056927a2177a705787c | HPC2N | nova-hpc2n | compute
> |
>
> | 52bb09199ec84ef09642151348eab695 | HPC2N | cinder | volume
> |
>
> | ac522bcd576b4d2b9f5adfb5405730be | regionOne | keystone | identity
> |
>
> | bad524f18fd74ec3b7fb6647ea661686 | regionOne | glance | image
> |
>
> +----------------------------------+-----------+--------------+--------------+
>
>
>
> But different names are working perfectly fine with glance and nova.
>
> Anyways thanks for all your time and efforts. If I manage to reproduce it
> on devstack, will report you back.
>
> Regards..
> Salman
>
>
>
>
> ------------------------------
> *From:* Joe Topjian [joe at topjian.net]
> *Sent:* Friday, December 11, 2015 7:16 PM
> *To:* Salman Toor
> *Cc:* openstack-operators at lists.openstack.org
> *Subject:* Re: [Openstack-operators] Cinder API with multiple regions not
> working.
>
> Hi Salman,
>
> I'm stumped.
>
> I was able to confirm that Keystone acts a little differently when you
> have multiple regional endpoints connected to the same "service_name" and
> "service_type" than if you have multiple regional endpoints, each with
> their own "service_name" but same "service_type"'s.
>
> For example, you have:
>
> regionOne, cinder, volume
> regionOne, cinderv2, volumev2
> HPC2N, cinderhpc2n, volume
> HPC2N, cinderv2hpc2n, volumev2
>
> When you do "openstack endpoint show volume", you should not get any
> output. But if you do "openstack endpoint show cinderhpc2n", then you will
> see output. Swap the service_type and service_names and the pattern
> continues.
>
> However, I am still unable to reproduce your issue.
>
> My initial tests used the following format:
>
> regionOne, cinder, volume
> regionOne, cinderv2, volumev2
> HPC2N, cinder, volume
> HPC2N, cinderv2, volumev2
>
> With this format, you will run into the UUID ordering issue I described
> previously. But again, I'm not able to reproduce the error you're seeing.
>
> I really hate to give this answer, but at this point, I'd recommend
> setting up a devstack environment and comparing the configuration files
> under /etc/nova and /etc/cinder. This problem is not occurring for me in
> devstack, so there has to be some difference between the two that is
> causing this issue.
>
> Once you have devstack installed, you can create a fictional second region
> by doing:
>
> Your current style:
>
> openstack service create --name cinderhpc2n volume
> openstack service create --name cinderhpc2nv2 volumev2
> openstack endpoint create --region HPC2N cinderhpc2n --publicurl
> http://10.1.0.113:8776/v1/%\(tenant_id\)s --internalurl
> http://10.1.0.113:8776/v1/%\(tenant_id\)s --adminurl
> http://10.1.0.113:8776/v1/%\(tenant_id\)s
> openstack endpoint create --region HPC2N cinderhpc2nv2 --publicurl
> http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl
> http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl
> http://10.1.0.113:8776/v2/%\(tenant_id\)s
>
> My style which I was originally testing with:
>
> openstack endpoint create --region HPC2N volume --publicurl
> http://10.1.0.113:8776/v1/%\(tenant_id\)s --internalurl
> http://10.1.0.113:8776/v1/%\(tenant_id\)s --adminurl
> http://10.1.0.113:8776/v1/%\(tenant_id\)s
> openstack endpoint create --region HPC2N volumev2 --publicurl
> http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl
> http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl
> http://10.1.0.113:8776/v2/%\(tenant_id\)s
>
> If the IP address is not in use, you will see commands hang when they try
> to reach the wrong region. I find this to be an easy troubleshooting method
> to discover where problems are being hit.
>
> If you figure out what the issue is, please let me know as I'm very
> curious.
>
> Joe
>
> On Fri, Dec 11, 2015 at 3:58 AM, Salman Toor <salman.toor at it.uu.se> wrote:
>
>> Hi,
>>
>> Still no luck. I have remove the “-“ from the names.
>>
>> *[root at controller: ~/openstack]* # openstack service list
>> +----------------------------------+---------------+--------------+
>> | ID | Name | Type |
>> +----------------------------------+---------------+--------------+
>> | 0f6736013cba469b9ddf21988a071952 | cinderhpc2n | volume |
>> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute |
>> | 2c7828120c294d3f82e3a17835babb85 | neutron | network |
>> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute |
>> | 478eff4e96464ae8a958ba29f750b14c | glance | image |
>> | 61d8baeb4ee74c7798a60758b2f4171f | cinderv2 | volumev2 |
>> | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | cinder | volume |
>> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image |
>> | dccd39b92ab547ddaf9047b38620145a | swift | object-store |
>> | dec231e13a114365b6e094421204255f | cinderhpc2nv2 | volumev2 |
>> | ebb1660d1d9746759a48de921521bfad | keystone | identity |
>> +----------------------------------+---------------+--------------+
>> *[root at controller: ~/openstack]* # openstack endpoint list
>>
>> +----------------------------------+-----------+---------------+--------------+
>> | ID | Region | Service Name | Service
>> Type |
>>
>> +----------------------------------+-----------+---------------+--------------+
>> | 3000cb23c6ab4ee5b68876ee08257338 | regionOne | cinderv2 |
>> volumev2 |
>> | e21e935ed861484e976d1b93e0fda0f0 | regionOne | nova | compute
>> |
>> | 42211e6e977d43308a4e288e68b97263 | HPC2N | cinderhpc2nv2 |
>> volumev2 |
>> | cdfb008c47a0472ab3a93e6ee07e9ba4 | regionOne | neutron | network
>> |
>> | e2693bcaf3da4be2810f04acd7995d7f | regionOne | cinder | volume
>> |
>> | 04acd1666704433d991cf9a75957c815 | HPC2N | glance-hpc2n | image
>> |
>> | eb1a12119c5c4b2dba78927982121be7 | HPC2N | cinderhpc2n | volume
>> |
>> | cfd1c766c56744309767cd84034f9bfb | regionOne | swift |
>> object-store |
>> | abe46f0d86064056927a2177a705787c | HPC2N | nova-hpc2n | compute
>> |
>> | ac522bcd576b4d2b9f5adfb5405730be | regionOne | keystone |
>> identity |
>> | bad524f18fd74ec3b7fb6647ea661686 | regionOne | glance | image
>> |
>>
>> +----------------------------------+-----------+---------------+--------------+
>> *[root at controller: ~/openstack]* # systemctl restart
>> openstack-nova-api.service
>>
>> Even with all the services didn’t help.
>>
>> *[root at controller: ~/openstack]* # nova volume-attach
>> 800d8ba0-cc17-4877-894a-89adecfb5eb7 85ab8b8a-c75c-45a1-9f51-44ed75ba3210
>> /dev/vdb
>> ERROR (ClientException): The server has either erred or is incapable of
>> performing the requested operation. (HTTP 500) (Request-ID:
>> req-f0d9372d-863b-43f6-8281-5c9196a866af)
>> *[root at controller: ~/openstack]* # openstack service delete cinderhpc2n
>> *[root at controller: ~/openstack]* # openstack service delete cinderhpc2nv2
>> *[root at controller: ~/openstack]* #
>> *[root at controller: ~/openstack]* #
>> *[root at controller: ~/openstack]* #
>> *[root at controller: ~/openstack]* #
>> *[root at controller: ~/openstack]* # nova volume-attach
>> 800d8ba0-cc17-4877-894a-89adecfb5eb7 85ab8b8a-c75c-45a1-9f51-44ed75ba3210
>> /dev/vdb
>> +----------+--------------------------------------+
>> | Property | Value |
>> +----------+--------------------------------------+
>> | device | /dev/vdb |
>> | id | 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 |
>> | serverId | 800d8ba0-cc17-4877-894a-89adecfb5eb7 |
>> | volumeId | 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 |
>>
>> +----------+--------------------------------------+
>>
>>
>> —— nova-api.log ——
>> 2015-12-11 09:43:08.534 16529 TRACE nova.api.openstack EndpointNotFound:
>> internalURL endpoint for volume service named cinder in regionOne region
>> not found
>> ——
>>
>> ——nova.conf
>> [cinder]
>> os_region_name = regionOne
>> ——
>>
>> —— cinder.conf
>> [default]
>> os_region_name = regionOne
>> ——
>>
>> Regards..
>> Salman.
>>
>>
>>
>>
>>
>> PhD, Scientific Computing
>> Researcher, IT Department,
>> Uppsala University.
>> Senior Cloud Architect,
>> SNIC.
>> Cloud Application Expert,
>> UPPMAX.
>> Visiting Researcher,
>> Helsinki Institute of Physics (HIP).
>> salman.toor at it.uu.se
>> http://www.it.uu.se/katalog/salto690
>>
>> On 10 Dec 2015, at 23:35, Joe Topjian <joe at topjian.net> wrote:
>>
>> Hi Salman,
>>
>> This has turned into a bit of fun -- I'm seeing a lot of wacky things.
>>
>> First, I'm pretty this issue is local to multi-regions and doesn't have
>> to do with having both Cinder v1 and v2 in the Keystone catalog. I changed
>> my catalog to only have Cinder v2 and I still see multi-region issues. If I
>> had more time, I would figure out how to forcefully make nova-api use
>> Cinder v1 to confirm it's not a v2 issue, but I'm pretty confident that it
>> is not.
>>
>> Second, strange things happen depending on the UUID of the endpoints.
>> Let's say I create two cinder v2 regions:
>>
>> openstack endpoint create --region RegionOne volumev2 --publicurl
>> http://10.1.0.112:8776/v2/%\(tenant_id\)s --internalurl
>> http://10.1.0.112:8776/v2/%\(tenant_id\)s --adminurl
>> http://10.1.0.112:8776/v2/%\(tenant_id\)s
>>
>> +--------------+-----------------------------------------+
>> | Field | Value |
>> +--------------+-----------------------------------------+
>> | adminurl | http://10.1.0.112:8776/v2/%(tenant_id)s |
>> | id | a46a5f86b0134944b66c25a7802f7b32 |
>> | internalurl | http://10.1.0.112:8776/v2/%(tenant_id)s |
>> | publicurl | http://10.1.0.112:8776/v2/%(tenant_id)s |
>> | region | RegionOne |
>> | service_id | 9cb6eeba4ae5484080ef1a5272b03367 |
>> | service_name | cinderv2 |
>> | service_type | volumev2 |
>> +--------------+-----------------------------------------+
>>
>> openstack endpoint create --region RegionTwo volumev2 --publicurl
>> http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl
>> http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl
>> http://10.1.0.113:8776/v2/%\(tenant_id\)s
>>
>> +--------------+-----------------------------------------+
>> | Field | Value |
>> +--------------+-----------------------------------------+
>> | adminurl | http://10.1.0.113:8776/v2/%(tenant_id)s |
>> | id | e9cd7a3fd8734b12a77154d73990261d |
>> | internalurl | http://10.1.0.113:8776/v2/%(tenant_id)s |
>> | publicurl | http://10.1.0.113:8776/v2/%(tenant_id)s |
>> | region | RegionTwo |
>> | service_id | 9cb6eeba4ae5484080ef1a5272b03367 |
>> | service_name | cinderv2 |
>> | service_type | volumev2 |
>> +--------------+-----------------------------------------+
>>
>> Note the IDs: RegionOne has a46... and RegionTwo has e9c...
>>
>> When I do "openstack endpoint show cinderv2", RegionTwo will be
>> displayed. Why? Because "e" comes after "a". If I regenerate the endpoints
>> until RegionOne has a UUID that is alphabetically later than RegionTwo,
>> then RegionOne will be displayed.
>>
>> It does not matter what my OS_REGION_NAME environment variable is set to,
>> either.
>>
>> Now, let's say my nova.conf file for the nova-api server in RegionOne
>> does not have the
>>
>> [cinder]
>> os_region_name = RegionOne
>>
>> setting I mentioned earlier (also, note that this needs to go in a
>> "[cinder]" section).
>>
>> If RegionOne's UUID comes earlier in the alphabet (so RegionOne would NOT
>> be displayed in "openstack endpoint show cinderv2"), then "nova
>> volume-attach" works. nova-api will grab the correct endpoint for RegionOne.
>>
>> If RegionOne's UUID comes later in the alphabet, then "nova
>> volume-attach" will try to contact RegionTwo.
>>
>> My guess is that the openstack client is reporting the last returned
>> catalog entry while the nova-api service is using the first returned
>> catalog entry.
>>
>> BUT, if I set
>>
>> [cinder]
>> os_region_name = RegionOne
>>
>> then "nova volume-attach" works, no matter what order the entries are in
>> the catalog.
>>
>>
>> Given all of the above, here is what is working for me:
>>
>> 1. I have two services in my Keystone catalog:
>>
>> cinder: volume
>> cinderv2: volumev2
>>
>> It looks like you have 4:
>>
>> cinder: volume
>> cinderv2: volumev2
>> cinder-hpc2n: volume
>> cinderv2-hpc2n: volumev2
>>
>> I don't know if that makes a difference, but maybe simplify the services
>> to just the standard 2.
>>
>> 2. nova.conf on the nova-api server has the "os_region_name" setting set
>> to the correct local region. I only had to restart the nova-api service to
>> see changes take effect, but maybe restart nova-* -- who knows at this
>> point!
>>
>> 3. Check the spelling of your regions. I noticed that you sometimes have
>> "regionOne" and other times have "RegionOne" but maybe that is just a typo
>> from sanitizing your data.
>>
>> Let me know if this helps :)
>>
>> Joe
>>
>> On Thu, Dec 10, 2015 at 2:31 AM, Salman Toor <salman.toor at it.uu.se>
>> wrote:
>>
>>> Hi Joe,
>>>
>>> Thanks for your reply, yes Me and Marco are working together.
>>>
>>> I have tried to used
>>>
>>> os_region_name = RegionOne
>>>
>>> both in cinder.conf and nova.conf but no luck so far. But the error
>>> message is a bit different. It seems like now the system is recognising the
>>> region.
>>>
>>>
>>> 2015-12-10 09:32:54.661 26792 TRACE nova.api.openstack EndpointNotFound:
>>> internalURL endpoint for volume service named cinder in regionOne region
>>> not found
>>>
>>> Following are my endpoints:
>>>
>>> ——
>>>
>>> *[root at controller: ~]* # openstack endpoint show cinder
>>>
>>> +--------------+------------------------------------------------+
>>>
>>> | Field | Value |
>>>
>>> +--------------+------------------------------------------------+
>>>
>>> | adminurl | http://cinder:8776/v1/%(tenant_id)s |
>>>
>>> | enabled | True |
>>>
>>> | id | e2693bcaf3da4be2810f04acd7995d7f |
>>>
>>> | internalurl | http://cinder:8776/v1/%(tenant_id)s |
>>>
>>> | publicurl | http://cinder:8776/v1/%(tenant_id)s |
>>>
>>> | region | regionOne |
>>>
>>> | service_id | 7bd5d667ec4b4d65b6c1b0de8b303fe3 |
>>>
>>> | service_name | cinder |
>>>
>>> | service_type | volume |
>>>
>>> +--------------+------------------------------------------------+
>>>
>>> *[root at controller: ~]* # openstack endpoint show cinderv2
>>>
>>> +--------------+------------------------------------------------+
>>>
>>> | Field | Value |
>>>
>>> +--------------+------------------------------------------------+
>>>
>>> | adminurl | http://cinder:8776/v2/%(tenant_id)s |
>>>
>>> | enabled | True |
>>>
>>> | id | 3000cb23c6ab4ee5b68876ee08257338 |
>>>
>>> | internalurl | http://cinder:8776/v2/%(tenant_id)s |
>>>
>>> | publicurl | http://cinder:8776/v2/%(tenant_id)s |
>>>
>>> | region | regionOne |
>>>
>>> | service_id | 61d8baeb4ee74c7798a60758b2f4171f |
>>>
>>> | service_name | cinderv2 |
>>>
>>> | service_type | volumev2 |
>>>
>>> +--------------+------------------------------------------------+
>>>
>>> *[root at controller: ~]* # openstack endpoint show cinder-hpc2n
>>>
>>> +--------------+--------------------------------------------------+
>>>
>>> | Field | Value |
>>>
>>> +--------------+--------------------------------------------------+
>>>
>>> | adminurl | http://cinder:8776/v1/%(tenant_id)s |
>>>
>>> | enabled | True |
>>>
>>> | id | c088f1f9ca0248a08eb60d4ac8492150 |
>>>
>>> | internalurl | http://cinder:8776/v1/%(tenant_id)s |
>>>
>>> | publicurl | http://cinder:8776/v1/%(tenant_id)s |
>>>
>>> | region | HPC2N |
>>>
>>> | service_id | 473f3eab043f4130b30d9ee64fe43703 |
>>>
>>> | service_name | cinder-hpc2n |
>>>
>>> | service_type | volume |
>>>
>>> +--------------+--------------------------------------------------+
>>>
>>> *[root at controller: ~]* # openstack endpoint show cinderv2-hpc2n
>>>
>>> +--------------+--------------------------------------------------+
>>>
>>> | Field | Value |
>>>
>>> +--------------+--------------------------------------------------+
>>>
>>> | adminurl | http://cinder:8776/v2/%(tenant_id)s |
>>>
>>> | enabled | True |
>>>
>>> | id | 985e5b8b068e4c3d99c12788236ee0fe |
>>>
>>> | internalurl | http://cinder:8776/v2/%(tenant_id)s |
>>>
>>> | publicurl | http://cinder:8776/v2/%(tenant_id)s |
>>>
>>> | region | HPC2N |
>>>
>>> | service_id | 089186fad4d7444b95e74ea9d7769423 |
>>>
>>> | service_name | cinderv2-hpc2n |
>>>
>>> | service_type | volumev2 |
>>>
>>>
>>> +--------------+--------------------------------------------------+
>>> ——
>>>
>>> Again if I remove hpc2n everything started to work fine. Can you tell me
>>> what service I suppose to restart (if required)?
>>>
>>> Regards..
>>> Salman
>>>
>>>
>>>
>>> PhD, Scientific Computing
>>> Researcher, IT Department,
>>> Uppsala University.
>>> Senior Cloud Architect,
>>> SNIC.
>>> Cloud Application Expert,
>>> UPPMAX.
>>> Visiting Researcher,
>>> Helsinki Institute of Physics (HIP).
>>> salman.toor at it.uu.se
>>> http://www.it.uu.se/katalog/salto690
>>>
>>> On 09 Dec 2015, at 17:48, Joe Topjian <joe at topjian.net> wrote:
>>>
>>> Hi Salman,
>>>
>>> Someone mentioned this same issue yesterday in relation to Terraform
>>> (maybe a colleague of yours?), so given the two occurrences, I thought I'd
>>> look into this.
>>>
>>> I have a Liberty environment readily available, so I created a second
>>> set of volume and volumev2 endpoints for a fictional region. Everything
>>> worked as expected, so I started reviewing the config files and saw that
>>> /etc/cinder/cinder.conf had an option
>>>
>>> [DEFAULT]
>>> os_region_name = RegionOne
>>>
>>> I commented that out, but things still worked.
>>>
>>> Then in /etc/nova/nova.conf, I saw:
>>>
>>> [cinder]
>>> os_region_name = RegionOne
>>>
>>> commenting this out caused volume attachments to hang indefinitely
>>> because nova was trying to contact cinder at RegionTwo (I'm assuming this
>>> is the first catalog entry that was returned).
>>>
>>> Given this is a Liberty environment, it's not accurately reproducing
>>> your problem, but could you check and see if you have that option set in
>>> nova.conf?
>>>
>>> I have a Kilo environment in the process of building. Once it has
>>> finished, I'll see if I can reproduce your error there.
>>>
>>> Thanks,
>>> Joe
>>>
>>> On Wed, Dec 9, 2015 at 4:35 AM, Salman Toor <salman.toor at it.uu.se>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I am using Kilo release on CentOS. We have recently enabled multiple
>>>> regions and it seems that Cinder have some problems with multiple
>>>> endpoints.
>>>>
>>>> Thinks are working fine with nova but cinder is behaving strange. Here
>>>> are my endpoints
>>>>
>>>>
>>>> ————
>>>> *[root at controller: ~]* # openstack service list
>>>> +----------------------------------+----------------+--------------+
>>>> | ID | Name | Type |
>>>> +----------------------------------+----------------+--------------+
>>>> | 0a33e6f259794ff2a99e626be37c0c2b | cinderv2-hpc2n | volumev2 |
>>>> | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute |
>>>> | 2c7828120c294d3f82e3a17835babb85 | neutron | network |
>>>> | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute |
>>>> | 478eff4e96464ae8a958ba29f750b14c | glance | image |
>>>> | 4a5a771d915e43c28e66538b8bc6e625 | cinder | volume |
>>>> | 72d1be82b2e5478dbf0f3fb9e7ba969d | cinderv2 | volumev2 |
>>>> | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image |
>>>> | a985795b49e2440db82970b81248c86e | cinder-hpc2n | volume |
>>>> | dccd39b92ab547ddaf9047b38620145a | swift | object-store |
>>>> | ebb1660d1d9746759a48de921521bfad | keystone | identity |
>>>> +----------------------------------+----------------+--------------+
>>>>
>>>> *[root at controller: ~]* # openstack endpoint
>>>> show a985795b49e2440db82970b81248c86e
>>>> +--------------+--------------------------------------------------+
>>>> | Field | Value |
>>>> +--------------+--------------------------------------------------+
>>>> | adminurl | http://<URL>:8776/v1/%(tenant_id)s |
>>>> | enabled | True |
>>>> | id | d4003e91ddf24cfb9fa497da81b01a18 |
>>>> | internalurl | http://<URL>:8776/v1/%(tenant_id)s |
>>>> | publicurl | http://<URL>:8776/v1/%(tenant_id)s |
>>>> | region | HPC2N |
>>>> | service_id | a985795b49e2440db82970b81248c86e |
>>>> | service_name | cinder-hpc2n |
>>>> | service_type | volume |
>>>> +--------------+--------------------------------------------------+
>>>>
>>>> *[root at controller: ~]* # openstack endpoint
>>>> show 4a5a771d915e43c28e66538b8bc6e625
>>>> +--------------+------------------------------------------------+
>>>> | Field | Value |
>>>> +--------------+------------------------------------------------+
>>>> | adminurl | http://<URL>:8776/v1/%(tenant_id)s |
>>>> | enabled | True |
>>>> | id | 5f19c0b535674dbd9e318c7b6d61b3bc |
>>>> | internalurl | http://<URL>:8776/v1/%(tenant_id)s |
>>>> | publicurl | http://<URL>:8776/v1/%(tenant_id)s |
>>>> | region | regionOne |
>>>> | service_id | 4a5a771d915e43c28e66538b8bc6e625 |
>>>> | service_name | cinder |
>>>> | service_type | volume |
>>>> +--------------+------------------------------------------------+
>>>>
>>>> And same for v2 endpoints
>>>>
>>>> ————
>>>>
>>>> ——— nova-api.log ———
>>>>
>>>> achmentController object at 0x598a3d0>>, body: {"volumeAttachment":
>>>> {"device": "", "volumeId": "93d96eab-e3fd-4131-9549-ed51e7299da2"}}
>>>> _process_stack
>>>> /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:780
>>>> 2015-12-08 12:52:05.847 3376 INFO
>>>> nova.api.openstack.compute.contrib.volumes
>>>> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
>>>> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Attach volume
>>>> 93d96eab-e3fd-4131-9549-ed51e7299da2 to instance
>>>> 3a4c8722-52a7-48f2-beb7-db8938698a0d at
>>>> 2015-12-08 12:52:05.847 3376 DEBUG nova.compute.api
>>>> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
>>>> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] [instance:
>>>> 3a4c8722-52a7-48f2-beb7-db8938698a0d] Fetching instance by UUID get
>>>> /usr/lib/python2.7/site-packages/nova/compute/api.py:1911
>>>> 2015-12-08 12:52:05.988 3376 ERROR nova.api.openstack
>>>> [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
>>>> 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Caught error: internalURL endpoint
>>>> for volume service named cinder not found
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack Traceback (most
>>>> recent call last):
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line
>>>> 125, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> req.get_response(self.application)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack application,
>>>> catch_exc_info=False)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in
>>>> call_application
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack app_iter =
>>>> application(self.environ, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> resp(environ, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
>>>> line 634, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> self._call_app(env, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
>>>> line 554, in _call_app
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> self._app(env, _fake_start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> resp(environ, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> resp(environ, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in
>>>> __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack response =
>>>> self.app(environ, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> resp(environ, start_response)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack resp =
>>>> self.call_func(req, *args, **self.kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> self.func(req, *args, **kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in
>>>> __call__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack content_type,
>>>> body, accept)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in
>>>> _process_stack
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack action_result
>>>> = self.dispatch(meth, request, action_args)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in
>>>> dispatch
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> method(req=request, **action_args)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/volumes.py",
>>>> line 305, in create
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_id,
>>>> device)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 230, in wrapped
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> func(self, context, target, *args, **kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 219, in inner
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> function(self, context, instance, *args, **kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 200, in inner
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> f(self, context, instance, *args, **kw)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3035, in
>>>> attach_volume
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack disk_bus,
>>>> device_type)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3016, in
>>>> _attach_volume
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
>>>> volume_bdm.destroy()
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in
>>>> __exit__
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
>>>> six.reraise(self.type_, self.value, self.tb)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3009, in
>>>> _attach_volume
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume =
>>>> self.volume_api.get(context, volume_id)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in
>>>> wrapper
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack res =
>>>> method(self, ctx, volume_id, *args, **kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 286, in get
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack item =
>>>> cinderclient(context).volumes.get(volume_id)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 115, in
>>>> cinderclient
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack url =
>>>> _SESSION.get_endpoint(auth, **service_parameters)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in
>>>> get_endpoint
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> auth.get_endpoint(self, **kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py",
>>>> line 219, in get_endpoint
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
>>>> service_name=service_name)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in
>>>> inner
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
>>>> func(*args, **kwargs)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
>>>> "/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line
>>>> 254, in url_for
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack raise
>>>> exceptions.EndpointNotFound(msg)
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack EndpointNotFound:
>>>> internalURL endpoint for volume service named cinder not found
>>>> 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
>>>>
>>>> ———
>>>>
>>>> If I remove cinder-hpc2n and cinderv2-hpc2n from the service list,
>>>> everything started to work fine.
>>>>
>>>> This is how I have created endpoints:
>>>>
>>>>
>>>> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s
>>>> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl http://<URL>:8776/v1/%\(tenant_id\)s
>>>> --region regionOne 4a5a771d915e43c28e66538b8bc6e625
>>>>
>>>> openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s
>>>> --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl http://<URL>:8776/v1/%\(tenant_id\)s
>>>> --region HPC2N a985795b49e2440db82970b81248c86e
>>>>
>>>> using the service-id.
>>>>
>>>>
>>>> Anyone experiencing this behavior? or Any suggestion how to fix it?
>>>> Again regions are working fine with nova and glance.
>>>>
>>>>
>>>> Regards..
>>>> Salman.
>>>>
>>>>
>>>> PhD, Scientific Computing
>>>> Researcher, IT Department,
>>>> Uppsala University.
>>>> Senior Cloud Architect,
>>>> SNIC.
>>>> Cloud Application Expert,
>>>> UPPMAX.
>>>> Visiting Researcher,
>>>> Helsinki Institute of Physics (HIP).
>>>> salman.toor at it.uu.se
>>>> http://www.it.uu.se/katalog/salto690
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-operators mailing list
>>>> OpenStack-operators at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>
>>>>
>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20151211/6610c4ab/attachment-0001.html>
More information about the OpenStack-operators
mailing list