Open Stack

On 09/25/2015 12:30 PM, Mitsuhiro Tanino wrote:
> On 09/22/2015 06:43 PM, Robert Collins wrote:
>> On 23 September 2015 at 09:52, Chris Friesen
>> <chris.friesen at windriver.com> wrote:
>>> Hi,
>>>
>>> I recently had an issue with one file out of a dozen or so in
>>> "/opt/cgcs/cinder/data/volumes/" being present but of size zero.  I'm
>>> running stable/kilo if it makes a difference.
>>>
>>> Looking at the code in
>>> volume.targets.tgt.TgtAdm.create_iscsi_target(), I'm wondering if we
>>> should do a fsync() before the close().  The way it stands now, it
>>> seems like it might be possible to write the file, start making use
>>> of it, and then take a power outage before it actually gets written
>>> to persistent storage.  When we come back up we could have an
>>> instance expecting to make use of it, but no target information in the on-disk copy of the file.
>
> I think even if there is no target information in configuration file dir, c-vol started successfully
> and iSCSI targets were created automatically and volumes were exported, right?
>
> There is an problem in this case that the iSCSI target was created without authentication because
> we can't get previous authentication from the configuration file.
>
> I'm curious what kind of problem did you met?

We had an issue in a private patch that was ported to Kilo without realizing 
that the data type of chap_auth had changed.

> In my understanding, the provider_auth in database has user name and password for iSCSI target.
> Therefore if we get authentication from DB, I think we can self-heal from this situation
> correctly after c-vol service is restarted.
>
> The lio target obtains authentication from provider_auth in database, but tgtd, iet, cxt obtain
> authentication from file to recreate iSCSI target when c-vol is restarted.
> If the file is missing, these volumes are exported without authentication and configuration
> file is recreated as I mentioned above.
>
> tgtd: Get target chap auth from file
> iet:  Get target chap auth from file
> cxt:  Get target chap auth from file
> lio:  Get target chap auth from Database(in provider_auth)
> scst: Get target chap auth by using original command
>
> If we get authentication from DB for tgtd, iet and cxt same as lio, we can recreate iSCSI target
> with proper authentication when c-vol is restarted.
> I think this is a solution for this situation.

If we fixed the chap auth info then we could live with a zero-size file. 
However, with the current code if we take a kernel panic or power outage it's 
theoretically possible to end up with a corrupt file of nonzero size (due to 
metadata hitting the persistant storage before the data).  I'm not confident 
that the current code would deal properly with that.

That said, if we always regenerate every file from the DB on cinder-volume 
startup (regardless of whether or not it existed, and without reading in the 
existing file), then we'd be okay without the robustness improvements.

Chris