[openstack-dev] [cinder] NFS mount as cinder user instead of root
Tom Barron
tpb at dyncloud.net
Mon Oct 19 12:13:18 UTC 2015
On 10/14/15 6:31 AM, Francesc Pinyol Margalef wrote:
> Hi,
> Yes, that worked! Thanks! :)
>
> But the process is very slow (about half an hour to create a volume).
> I think the problem is the execution of "du -sb --apparent-size
> --exclude *snapshot*
> /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51", as shown in the logs:
>
> 2015-10-13 19:33:14.127 1311 INFO
> cinder.volume.flows.manager.create_volume
> [req-f52e5048-3155-4d49-92c0-4152b8243fd6
> 26e01a732d9e44d4a98305c6aa11860f 36593fc96ab64bc7959eb9e0ff2f2247 - - -]
> Volume 5230104d-68a3-4dc0-95ec-43f5d8fbc5d3: b
> eing created as raw with specification: {'status': u'creating',
> 'volume_size': 1, 'volume_name':
> u'volume-5230104d-68a3-4dc0-95ec-43f5d8fbc5d3'}
> 2015-10-13 19:33:14.140 1311 INFO cinder.brick.remotefs.remotefs
> [req-f52e5048-3155-4d49-92c0-4152b8243fd6
> 26e01a732d9e44d4a98305c6aa11860f 36593fc96ab64bc7959eb9e0ff2f2247 - - -]
> Already mounted: /var/lib/cinder/mnt/9ae799cf301b19940950
> ae49dd800c51
> 2015-10-13 19:40:27.556 1311 WARNING cinder.openstack.common.loopingcall
> [req-0a4a8e09-f10b-4dc6-96bf-f7e333635f99 - - - - -] task u'<bound
> method Service.periodic_tasks of <cinder.service.Service object at
> 0x2c5c650>>' run outlasted int
> erval by 499.80 sec
> 2015-10-13 19:40:27.564 1311 INFO cinder.volume.manager
> [req-5b14e3f3-76d9-484e-819b-46da8f0e29a6 - - - - -] Updating volume status
> 2015-10-13 19:40:27.577 1311 INFO cinder.brick.remotefs.remotefs
> [req-5b14e3f3-76d9-484e-819b-46da8f0e29a6 - - - - -] Already mounted:
> /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51
> 2015-10-13 19:51:37.371 1311 WARNING cinder.openstack.common.loopingcall
> [req-5b14e3f3-76d9-484e-819b-46da8f0e29a6 - - - - -] task u'<bound
> method Service.periodic_tasks of <cinder.service.Service object at
> 0x2c5c650>>' run outlasted int
> erval by 609.81 sec
> 2015-10-13 19:51:37.378 1311 INFO cinder.volume.manager
> [req-941c5a78-a85d-4fa8-9df9-033b4cc6e6f5 - - - - -] Updating volume status
> 2015-10-13 19:51:37.391 1311 INFO cinder.brick.remotefs.remotefs
> [req-941c5a78-a85d-4fa8-9df9-033b4cc6e6f5 - - - - -] Already mounted:
> /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51
> 2015-10-13 19:58:18.585 1311 ERROR cinder.openstack.common.periodic_task
> [req-941c5a78-a85d-4fa8-9df9-033b4cc6e6f5 - - - - -] Error during
> VolumeManager._report_driver_status: Unexpected error while running command.
> Command: None
> Exit code: -
> Stdout: u"Unexpected error while running command.\nCommand: du -sb
> --apparent-size --exclude *snapshot*
> /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51\nExit code:
> -15\nStdout: u''\nStderr: u''"
> Stderr: None
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> Traceback (most recent call last):
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/cinder/openstack/common/periodic_task.py",
> line 224, in run_periodic_tasks
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task task(self, context)
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/cinder/volume/manager.py", line 1499,
> in _report_driver_status
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task volume_stats =
> self.driver.get_volume_stats(refresh=True)
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in
> wrapper
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task return f(*args, **kwargs)
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py",
> line 439, in get_volume_stats
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task self._update_volume_stats()
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/remotefs.py",
> line 458, in _update_volume_stats
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task capacity, free, used =
> self._get_capacity_info(share)
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/cinder/volume/drivers/nfs.py", line
> 281, in _get_capacity_info
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task run_as_root=run_as_root)
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/cinder/utils.py", line 143, in execute
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task return
> processutils.execute(*cmd, **kwargs)
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task File
> "/usr/lib/python2.7/site-packages/oslo_concurrency/processutils.py",
> line 233, in execute
> 2015-10-13 19:58:18.585 1311 TRACE
> cinder.openstack.common.periodic_task cmd=sanitized_cmd)
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> ProcessExecutionError: Unexpected error while running command.
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> Command: du -sb --apparent-size --exclude *snapshot*
> /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> Exit code: -15
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> Stdout: u''
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> Stderr: u''
> 2015-10-13 19:58:18.585 1311 TRACE cinder.openstack.common.periodic_task
> 2015-10-13 19:58:18.593 1311 WARNING cinder.openstack.common.loopingcall
> [req-941c5a78-a85d-4fa8-9df9-033b4cc6e6f5 - - - - -] task u'<bound
> method Service.periodic_tasks of <cinder.service.Service object at
> 0x2c5c650>>' run outlasted int
> erval by 341.22 sec
> 2015-10-13 19:58:18.599 1311 INFO cinder.volume.manager
> [req-44535964-f7f8-4128-a7d3-5ab8c9c3ee41 - - - - -] Updating volume status
> 2015-10-13 19:58:18.613 1311 INFO cinder.brick.remotefs.remotefs
> [req-44535964-f7f8-4128-a7d3-5ab8c9c3ee41 - - - - -] Already mounted:
> /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51
> 2015-10-13 20:01:42.563 1311 INFO cinder.volume.drivers.remotefs
> [req-f52e5048-3155-4d49-92c0-4152b8243fd6
> 26e01a732d9e44d4a98305c6aa11860f 36593fc96ab64bc7959eb9e0ff2f2247 - - -]
> casted to 192.168.1.101:/mnt/nas/nv3/data/file/
> 2015-10-13 20:01:42.727 1311 INFO
> cinder.volume.flows.manager.create_volume
> [req-f52e5048-3155-4d49-92c0-4152b8243fd6
> 26e01a732d9e44d4a98305c6aa11860f 36593fc96ab64bc7959eb9e0ff2f2247 - - -]
> Volume volume-5230104d-68a3-4dc0-95ec-43f5d8fb
> c5d3 (5230104d-68a3-4dc0-95ec-43f5d8fbc5d3): created successfully
>
>
>
> The NFS server contains a lot of files, and even a plain execution, from
> command line,
> of "du /var/lib/cinder/mnt/9ae799cf301b19940950ae49dd800c51" takes more
> than 10 minutes.
>
> Can this execution of du be disabled? Why not using df?
>
> Francesc
Well, a 'stat' command is run to get the number of blocks free and
allocated on the backing filesystem, as well as the blocksize, in order
to calculate total space and free space on the NFS share. This is the
equivalent of a 'df' command.
The 'du' command that follows can indeed be slow because it examines
each file in the file system and sums up their "apparent" sizes in order
to determine the "allocated space".
See
https://github.com/openstack/cinder/blob/master/cinder/volume/drivers/nfs.py#L277L296.
Allocated space reflects the nominal sizes of the cinder volumes and
will not in general be equal to the difference between total space and
free space.
Consider an NFS share with three 100TB cinder volumes. Allocated space
will be 300TB (plus a bit for directories). But because these volumes
are by default created sparse, they will initially not consume any file
system blocks and free space will be very close to total space.
The NFS driver uses the amount of allocated space when deciding which of
several shares to use when creating a volume, or when deciding if it is
safe to extend a volume.
Now the cinder volume manager itself keeps track of space allocations
when volumes are created and passes this information to the scheduler
along with total capacity and free capacity as reported by volume
drivers. So one might ask why the NFS driver has to do this as well.
If the NFS driver reported each share to the scheduler as a separate
storage pool, or if each instance of the driver were limited to a single
share, then I think we could eliminate the need to track allocated space
in the driver itself because the scheduler would be making all share
placement decisions. But as things stand, the driver itself must select
among multiple shares and therefore needs to take the amount of
committed space into account, not just the amount currently consumed.
-- Tom
>
>
>
> 2015-10-03 14:00 GMT+02:00 <openstack-dev-request at lists.openstack.org
> <mailto:openstack-dev-request at lists.openstack.org>>:
>
> Message: 6
> Date: Fri, 2 Oct 2015 09:28:16 -0400
> From: Eric Harney <eharney at redhat.com <mailto:eharney at redhat.com>>
> To: "OpenStack Development Mailing List (not for usage questions)"
> <openstack-dev at lists.openstack.org
> <mailto:openstack-dev at lists.openstack.org>>
> Subject: Re: [openstack-dev] [cinder] NFS mount as cinder user instead
> of root
> Message-ID: <560E8670.5030603 at redhat.com
> <mailto:560E8670.5030603 at redhat.com>>
> Content-Type: text/plain; charset=windows-1252
>
> On 10/02/2015 05:48 AM, Francesc Pinyol Margalef wrote:
> > Hi,
> > In a previous message in general openstack list I reported a problem
> > when trying to mount an NFS volume from a Fujitsu Eternus DX
> > http://lists.openstack.org/pipermail/openstack/2015-July/013578.html
> >
> > The issue is that this file server does not allow mounts as root, and
> > this behaviour cannot be changed.
> >
> > Would it be possible to configure (or modify) Cinder in order to mount
> > NFS points as cinder user instead of root?
> >
> > Francesc
> >
> >
> >
>
> With the NFS driver, setting the option "nas_secure_file_operations =
> True" should cause files to be created as the cinder user rather than as
> root.
>
> Can you try setting this and let me know if that helps?
>
> Eric
>
>
>
>
>
> --
> Francesc Pinyol Margalef
> http://www.francescpinyol.cat/
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list