[Openstack] glance image service: 500 Internal Server Error

Tom Walsh expresswebsys+openstack at gmail.com
Fri Nov 13 19:34:58 UTC 2015


Joerg,

The logs indicate that you have a problem connecting to MySQL. Verify that
you are using the correct credentials to connect to the glance database. I
am not sure if this a typo or just you modifying information from the
configs to sanitize them, but you have:

connection = mysql://glance:xxxxxx@controller/glance

But the logs complain about:

Access denied for user 'glance'@'localhost'

Is your MySQL instance on the localhost?

It might be possible that you are not using the correct credentials. Our
Liberty setup uses a separate section for the database:

[database]
connection = mysql+pymysql://glance:password@192.168.254.1/glance
backend = sqlalchemy

Double check your connectivity using the mysql CLI and passing in the same
variables you are using in your connection string:

mysql -h 192.168.254.1 -u glance -p

See if that works. If it does, start working your way up from there as you
know you have the right credentials as far as MySQL is concerned and start
looking elsewhere.

I suspect it might be the configuration of glance, but that's just a guess
at this point.

Also one thing that bit us was max_connections in MySQL. We had working
details, but MySQL was preventing additional connections due to being out
of connection slots. Just mentioning it that there can be other reasons
MYSQL might reject a connection other than username/password.

Good luck.

Tom Walsh
ExpressHosting.net


On Fri, Nov 13, 2015 at 12:11 PM, Joerg Streckfuss <openstack at dirtyhack.org>
wrote:

> Dear list,
>
> i tried to install the image service on centos7 from openstack liberty.
> When i try to create an image i got the folliwing error:
>
> <snip>
> # glance image-create --name "cirros" \
>   --file cirros-0.3.4-x86_64-disk.img  \
>   --disk-format qcow2 --container-format bare \
>   --visibility public --progress
> 500 Internal Server Error: The server has either erred or is incapable of
> performing the requested operation. (HTTP 500)
> <snap>
>
> glance-api.conf:
> <snip>
> [DEFAULT]
> connection = mysql://glance:xxxxxx@controller/glance
> notification_driver = noop
> verbose = True
>
> [keystone_authtoken]
> auth_uri = http://controller:5000
> auth_url = http://controller:35357
> auth_plugin = password
> project_domain_id = default
> user_domain_id = default
> project_name = service
> username = glance
> password = xxxxxx
>
> [paste_deploy]
> flavor = keystone
>
> [glance_store]
> default_store = file
> filesystem_store_datadir = /var/lib/glance/images/
> <snap>
>
>
> openstack-status gives me the folling output
>
> <snip>
> # openstack-status
> == Glance services ==
> openstack-glance-api:                   active
> openstack-glance-registry:              active
> == Keystone service ==
> openstack-keystone:                     inactive  (disabled on boot)
> == Support services ==
> mysqld:                                 inactive  (disabled on boot)
> dbus:                                   active
> rabbitmq-server:                        active
> memcached:                              active
> == Keystone users ==
> /usr/lib/python2.7/site-packages/keystoneclient/shell.py:64:
> DeprecationWarning: The keystone CLI is deprecated in favor of
> python-openstackclient. For a Python library, continue using
> python-keystoneclient.
>   'python-keystoneclient.', DeprecationWarning)
> WARNING: unsupported identity-api-version 3, falling back to 2.0
> /usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145:
> DeprecationWarning: Constructing an instance of the
> keystoneclient.v2_0.client.Client class without a session is deprecated as
> of the 1.7.0 release and may be removed in the 2.0.0 release.
>   'the 2.0.0 release.', DeprecationWarning)
> /usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147:
> DeprecationWarning: Using the 'tenant_name' argument is deprecated in
> version '1.7.0' and will be removed in version '2.0.0', please use the
> 'project_name' argument instead
>   super(Client, self).__init__(**kwargs)
> /usr/lib/python2.7/site-packages/debtcollector/renames.py:43:
> DeprecationWarning: Using the 'tenant_id' argument is deprecated in version
> '1.7.0' and will be removed in version '2.0.0', please use the 'project_id'
> argument instead
>   return f(*args, **kwargs)
> /usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:376:
> DeprecationWarning: Constructing an HTTPClient instance without using a
> session is deprecated as of the 1.7.0 release and may be removed in the
> 2.0.0 release.
>   'the 2.0.0 release.', DeprecationWarning)
> Authorization Failed: The resource could not be found. (HTTP 404)
> (Request-ID: req-99d108ad-edb3-4589-a7c1-85dc8e97f63a)
> == Glance images ==
> 500 Internal Server Error: The server has either erred or is incapable of
> performing the requested operation. (HTTP 500)
> <snap>
>
> in /var/log/glance/api.log i got the folling exception
>
> <snip>
> 2015-11-13 18:00:18.383 1458 INFO eventlet.wsgi.server
> [req-7e4bee11-b8d7-4dcf-a0a0-b2e98cf3593c 058f089c31e64370af9cb268c75da386
> ff2149063f074c90bae7ea0f1dfcdfaf - - -] 10.11.12.230 - - [13/Nov/2015
> 18:00:18] "GET /v2/schemas/imag[45/89]
> 1.1" 200 4117 2.556846
> 2015-11-13 18:00:18.389 1458 INFO eventlet.wsgi.server
> [req-b0894879-3658-4bd7-ac23-73818a607628 058f089c31e64370af9cb268c75da386
> ff2149063f074c90bae7ea0f1dfcdfaf - - -] 10.11.12.230 - - [13/Nov/2015
> 18:00:18] "GET /v2/schemas/image HTTP/
> 1.1" 200 4117 0.003295
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> [req-76502a62-8c24-4b0a-828e-a0bf07733023 058f089c31e64370af9cb268c75da386
> ff2149063f074c90bae7ea0f1dfcdfaf - - -] Caught error:
> (_mysql_exceptions.OperationalError) (1045, "Access den
> ied for user 'glance'@'localhost' (using password: YES)")
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi Traceback (most
> recent call last):
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 879, in
> __call__
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     request,
> **action_args)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/common/wsgi.py", line 907, in
> dispatch
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> method(*args, **kwargs)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 504, in
> wrapped
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> func(self, req, *args, **kwargs)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/api/v2/images.py", line 66, in
> create
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi image_repo.add(image)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 94, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     result =
> self.base.add(base_item)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/notifier.py", line 460, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> super(ImageRepoProxy, self).add(image)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 94, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     result =
> self.base.add(base_item)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/api/policy.py", line 131, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> super(ImageRepoProxy, self).add(image)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 94, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     result =
> self.base.add(base_item)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/quota/__init__.py", line 114, in
> add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> super(ImageRepoProxy, self).add(image)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 94, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     result =
> self.base.add(base_item)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/location.py", line 63, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     result =
> super(ImageRepoProxy, self).add(image)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/domain/proxy.py", line 94, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     result =
> self.base.add(base_item)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/db/__init__.py", line 261, in add
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     new_values =
> self.db_api.image_create(self.context, image_values)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/api.py", line 128,
> in image_create
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> _image_update(context, values, None, purge_props=False)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/retrying.py", line 68, in wrapped_f
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> Retrying(*dargs, **dkw).call(f, *args, **kw)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/retrying.py", line 223, in call
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> attempt.get(self._wrap_exception)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/retrying.py", line 261, in get
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> six.reraise(self.value[0], self.value[1], self.value[2])
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/retrying.py", line 217, in call
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     attempt =
> Attempt(fn(*args, **kwargs), attempt_number, False)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/common/utils.py", line 697, in
> wrapper
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return f(*args,
> **kwargs)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/api.py", line 708,
> in _image_update
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     session =
> get_session()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/api.py", line 101,
> in get_session
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     facade =
> _create_facade_lazily()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/glance/db/sqlalchemy/api.py", line 86, in
> _create_facade_lazily
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     _FACADE =
> session.EngineFacade.from_config(CONF)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
> 1015, in from_config
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> expire_on_commit=expire_on_commit, _conf=conf)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
> 943, in __init__
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> slave_connection=slave_connection)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
> 338, in _start
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     engine_args,
> maker_args)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/enginefacade.py", line
> 362, in _setup_for_connection
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> sql_connection=sql_connection, **engine_kwargs)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 152,
> in create_engine
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     test_conn =
> _test_connection(engine, max_retries, retry_interval)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib/python2.7/site-packages/oslo_db/sqlalchemy/engines.py", line 326,
> in _test_connection
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> engine.connect()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2013,
> in connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> self._connection_cls(self, **kwargs)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 72, in
> __init__
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     if connection is
> not None else engine.raw_connection()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2099,
> in raw_connection
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> self.pool.unique_connection, _connection)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2073,
> in _wrap_pool_connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     e, dialect, self)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1399,
> in _handle_dbapi_exception_noconnection
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> util.raise_from_cause(newraise, exc_info)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 199,
> in raise_from_cause
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> reraise(type(exception), exception, tb=exc_tb)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 2069,
> in _wrap_pool_connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return fn()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 318, in
> unique_connection
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> _ConnectionFairy._checkout(self)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 708, in
> _checkout
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     fairy =
> _ConnectionRecord.checkout(pool)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 480, in
> checkout
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     rec =
> pool._do_get()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1049, in
> _do_get
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi self._dec_overflow()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line
> 60, in __exit__
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> compat.reraise(exc_type, exc_value, exc_tb)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 1046, in
> _do_get
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> self._create_connection()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 323, in
> _create_connection
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> _ConnectionRecord(self)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 449, in
> __init__
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi self.connection =
> self.__connect()
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/pool.py", line 602, in
> __connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     connection =
> self.__pool._invoke_creator(self)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/strategies.py", line
> 97, in connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> dialect.connect(*cargs, **cparams)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line
> 377, in connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> self.dbapi.connect(*cargs, **cparams)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/MySQLdb/__init__.py", line 81, in
> Connect
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi     return
> Connection(*args, **kwargs)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi   File
> "/usr/lib64/python2.7/site-packages/MySQLdb/connections.py", line 187, in
> __init__
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi super(Connection,
> self).__init__(*args, **kwargs2)
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi OperationalError:
> (_mysql_exceptions.OperationalError) (1045, "Access denied for user
> 'glance'@'localhost' (using password: YES)")
> 2015-11-13 18:00:18.511 1458 ERROR glance.common.wsgi
> 2015-11-13 18:00:18.688 1458 INFO eventlet.wsgi.server
> [req-76502a62-8c24-4b0a-828e-a0bf07733023 058f089c31e64370af9cb268c75da386
> ff2149063f074c90bae7ea0f1dfcdfaf - - -] 10.11.12.230 - - [13/Nov/2015
> 18:00:18] "POST /v2/images HTTP/1.1" 5
> 00 454 0.222921
> <snap>
>
>
> What does the entry 'ERROR glance.common.wsgi OperationalError:
> (_mysql_exceptions.OperationalError) (1045, "Access denied for user
> 'glance'@'localhost' (using password: YES)")' mean. Does it mean the
> password for the user glance in mysql does not match.
>
> I double checked it by setting it once more in the database but the error
> remains.
>
> Thanks,
>
> Joerg
>
> _______________________________________________
> Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe :
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20151113/d17bb486/attachment.html>


More information about the Openstack mailing list