Open Stack

Hi there, 

I’m having an issue enabling Glance scrubber in Icehouse:

$ service openstack-glance-scrubber status
openstack-glance-scrubber dead but subsys locked
I’ve tried:
- checking the PID, but there are no running processes found:
$ cat /var/run/glance/glance-scrubber.pid
2834
$ kill -9 2834
bash: kill: (2834) - No such process
- removing the lockfile and restarting the process, but it goes back to “dead but subsys locked” immediately:

$ service openstack-glance-scrubber status
openstack-glance-scrubber dead but subsys locked


My Glance scrubber config is as follows:

[DEFAULT]
# Show more verbose log output (sets INFO log level output)
verbose = True

# Show debugging output in logs (sets DEBUG log level output)
debug = False


# Log to this directory
log_dir=/var/log/glance

# Format string to use for log messages with context. (string
# value)
logging_context_format_string=%(asctime)s.%(msecs)03d %(levelname)s %(message)s pid[%(process)s] %(request_id)s %(instance)s
# Format string to use for log messages without context.
# (string value)
logging_default_format_string=%(asctime)s.%(msecs)03d %(levelname)s %(message)s pid[%(process)s]

# Data to append to log format when level is DEBUG. (string
# value)
logging_debug_format_suffix=%(funcName)s %(pathname)s:%(lineno)d

# Prefix each line of exception output with this format.
# (string value)
logging_exception_prefix=%(asctime)s.%(msecs)03d TRACE instance[%(instance)s]

# Should we run our own loop or rely on cron/scheduler to run us
daemon = False

# Loop time between checking the db for new items to schedule for delete
wakeup_time = 300

# Directory that the scrubber will use to remind itself of what to delete
# Make sure this is also set in glance-api.conf
scrubber_datadir = /data/var/lib/glance/scrubber

# Only one server in your deployment should be designated the cleanup host
cleanup_scrubber = False

# pending_delete items older than this time are candidates for cleanup
cleanup_scrubber_time = 86400

# Address to find the registry server for cleanups
registry_host = os-controller.wd5-cedev21.az2.eng.pdx.wd

# Port the registry server is listening on
registry_port = 9191

# ============ Filesystem Store Options ========================

# Directory that the Filesystem backend store
# writes image data to
filesystem_store_datadir = /data/var/lib/glance/images


When I attempted to launch “glance-scrubber”, the output to scrubber.log is as follows:

2016-06-27 13:34:36.953 WARNING Deprecated: glance.store.rbd.Store not found in `known_store`. Stores need to be explicitly enabled in the configuration file. pid[3983]
2016-06-27 13:34:36.954 WARNING Failed to configure store correctly: Store gridfs could not be configured correctly. Reason: Missing dependencies: pymongo Disabling add method. pid[3983]
2016-06-27 13:34:36.955 WARNING Deprecated: glance.store.gridfs.Store not found in `known_store`. Stores need to be explicitly enabled in the configuration file. pid[3983]
2016-06-27 13:34:36.993 WARNING Failed to configure store correctly: Store cinder could not be configured correctly. Reason: Cinder storage requires a context. Disabling add method. pid[3983]
2016-06-27 13:34:36.994 WARNING Deprecated: glance.store.cinder.Store not found in `known_store`. Stores need to be explicitly enabled in the configuration file. pid[3983]
2016-06-27 13:34:37.000 ERROR Could not find swift_store_auth_address in configuration options. pid[3983]
2016-06-27 13:34:37.001 WARNING Failed to configure store correctly: Store swift could not be configured correctly. Reason: Could not find swift_store_auth_address in configuration options. Disabling add method. pid[3983]
2016-06-27 13:34:37.001 WARNING Deprecated: glance.store.swift.Store not found in `known_store`. Stores need to be explicitly enabled in the configuration file. pid[3983]
2016-06-27 13:34:37.004 WARNING Failed to configure store correctly: Store s3 could not be configured correctly. Reason: Could not find s3_store_host in configuration options. Disabling add method. pid[3983]
2016-06-27 13:34:37.005 WARNING Deprecated: glance.store.s3.Store not found in `known_store`. Stores need to be explicitly enabled in the configuration file. pid[3983]
2016-06-27 13:34:37.007 INFO Initializing scrubber with configuration: {'cleanup_time': 86400, 'registry_host': 'os-controller.wd5-cedev21.az2.eng.pdx.wd', 'cleanup': False, 'scrubber_datadir': '/data/var/lib/glance/scrubber', 'registry_port': 9191} pid[3983]
Despite glance-scrubber not working, there is not a massive discrepancy between active glance images & those stored locally on the filesystem:

$ ls /data/var/lib/glance/images/ | wc -l
1094
$ glance image-list --all-tenants | grep active | wc -l
1083
My questions are:

- i) Based upon my config and stack trace, does anyone have any ideas on why the Glance scrubber process would be stuck in a "dead but subsys locked” state
- ii) If glance-scrubber is not working, are these images being cleaned up some other way (i.e. through Glance itself)?

Thanks.

-- mickm.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160627/5d7f5ca1/attachment.html>