Jonathan, James, A quick thank you - I wanted to report that I finally got it working. A lot of fiddling around with UID/GID settings on the NFS share. (Oddly, Cinder didn't need this.) One thing that had me going for a bit - I thought I'd test the NFS permissions outside of the glance-api process by using 'su -c "touch /var/lib/glance/images/testfile" glance'. When it didn't work, I assumed I didn't have NFS configured correctly. It turns out that the above command won't work if the shell for user glance is set to /bin/false. So, in the case of problems I would propose this as a debugging/verification technique - make sure that user glance can write to the NFS share by 'chsh -s /bin/bash glance' and then the above 'su -c' command. -Dave -- Dave Hall Binghamton University kdhall@binghamton.edu 607-760-2328 (Cell) 607-777-4641 (Office) On Wed, Aug 3, 2022 at 11:56 AM Jonathan Rosser < jonathan.rosser@rd.bbc.co.uk> wrote:

You should set this up like the example given previously by James Denton, with the variables set in user-variables.yml, not in openstack-user-config.yml.

For reference we run a CI job that tests glance+NFS and the configuration used in that is here https://github.com/openstack/openstack-ansible/blob/master/tests/roles/boots.... You would be able to reproduce that using an all-in-one setup by adding 'nfs' to the SCENARIO environment variable.

If you enable debug on a service and want to watch the log, you can use journalctl -fu <unit-name>, it should not be necessary to specifically run it in the foreground.

Hope this helps, Jonathan. On 03/08/2022 15:35, Dave Hall wrote:

Hello,

(Any help on this would be greatly appreciated. I've been chasing this for 2 weeks now...)

I posted the other day that I had solved this issue, but now it's back. The primary glance-api error message is:

Error in store configuration. Adding images to store is disabled.

So why is my store disabled? Is there some command or config line that controls this? Shouldn't the store be enabled by default?

The relevant stanza in my openstack-user-config.yml is

image_hosts: # infra38: # ip: 172.29.236.38 # container_vars: # limit_container_types: glance # glance_remote_client: # - what: "172.29.244.27:/images" # where: "/var/lib/glance/images" # type: "nfs" # options: "_netdev,vers=3,proto=tcp,sec=sys"

infra38: ip: 172.29.236.38 container_vars: glance_default_store: file glance_nfs_local_directory: "images" glance_nfs_client: - server: "172.29.244.27" remote_path: "/images" local_path: "/var/lib/glance/images" type: "nfs" options: "_netdev,vers=3,proto=tcp,sec=sys,noauto,user" config_overrides: "{}"

The stanza that's commented out produced the same result. My user_variables.yaml does not have any lines pertaining to glance. Also, I've removed the stanzas for the other 2 infra hosts.

The glance-api.conf file from the container is:

[DEFAULT] # Disable stderr logging use_stderr = False debug = False use_journal = True fatal_deprecations = False bind_host = 172.29.238.205 bind_port = 9292 http_keepalive = True digest_algorithm = sha256 backlog = 4096 workers = 16 cinder_catalog_info = volumev3:cinderv3:internalURL enable_v2_api = True transport_url = rabbit:// glance:9e7f205ef620983a542b4f915420e50d33f774@172.29.238.84:5671, glance:9e7f205ef620983a542b4f915420e50d33f774@172.29.238.174:5671, glance:9e7f205ef620983a542b4f915420e50d33f774@172.29.239.33:5671//glance?ssl=1&ssl_version=TLSv1_2&ssl_ca_file= scrub_time = 43200 image_cache_dir = /var/lib/glance/cache/ image_cache_stall_time = 86400 image_cache_max_size = 10737418240 # defaults to true if RBD is used as default store show_image_direct_url = False show_multiple_locations = True enabled_backends = file:file,http:http,cinder:cinder

[task] task_executor = taskflow

[database] connection = mysql+pymysql:// glance:e6dd6f2ca946c9e6f72bb864387a@172.29.236.36/glance?charset=utf8&ssl_verify_cert=true max_overflow = 50 max_pool_size = 5 pool_timeout = 30 connection_recycle_time = 600

[keystone_authtoken] insecure = False auth_type = password auth_url = http://172.29.236.36:5000/v3 www_authenticate_uri = http://172.29.236.36:5000 project_domain_id = default user_domain_id = default project_name = service username = glance password = c96a36e76208ee26851c78670d34dcaff1c870 region_name = RegionOne service_token_roles_required = False service_token_roles = service service_type = image memcached_servers = 172.29.239.168:11211,172.29.239.242:11211, 172.29.236.80:11211 token_cache_time = 300 # if your memcached server is shared, use these settings to avoid cache poisoning memcache_security_strategy = ENCRYPT memcache_secret_key = 0b65b4b99155a6430e923fc9c24d9674

[oslo_policy] policy_file = policy.yaml policy_default_rule = default policy_dirs = policy.d

[oslo_messaging_notifications] topics = notifications driver = messagingv2 transport_url = rabbit:// glance:9e7f205ef620983a542b4f915420e50d33f774@172.29.238.84:5671, glance:9e7f205ef620983a542b4f915420e50d33f774@172.29.238.174:5671, glance:9e7f205ef620983a542b4f915420e50d33f774@172.29.239.33:5671//glance?ssl=1&ssl_version=TLSv1_2&ssl_ca_file=

[paste_deploy] flavor = keystone+cachemanagement

[glance_store] default_backend = file

[file] filesystem_store_datadir = /var/lib/glance/images/

[profiler] enabled = False

[oslo_middleware] enable_proxy_headers_parsing = True

[cors] allow_headers = origin,content-md5,x-image-meta-checksum,x-storage-token,accept-encoding,x-auth-token,x-identity-status,x-roles,x-service-catalog,x-user-id,x-tenant-id,x-openstack-request-id allow_methods = GET,POST,PUT,PATCH,DELETE allowed_origin = https://osa-portal.cs.binghamton.edu

I'd be glad to run glance-api in the foreground with debug (in the container, or course), but it's not obvious from the .service file how to do that. I'd be glad to read source code, but a pointer or two would be handy. If NFS (or Hao's GPFS) just aren't supported anymore please tell me.

Thanks.

-Dave

-- Dave Hall Binghamton University kdhall@binghamton.edu