Open Stack

On 09/25/2015 02:30 PM, Mitsuhiro Tanino wrote:
> On 09/22/2015 06:43 PM, Robert Collins wrote:
>> On 23 September 2015 at 09:52, Chris Friesen 
>> <chris.friesen at windriver.com> wrote:
>>> Hi,
>>>
>>> I recently had an issue with one file out of a dozen or so in 
>>> "/opt/cgcs/cinder/data/volumes/" being present but of size zero.  I'm 
>>> running stable/kilo if it makes a difference.
>>>
>>> Looking at the code in 
>>> volume.targets.tgt.TgtAdm.create_iscsi_target(), I'm wondering if we 
>>> should do a fsync() before the close().  The way it stands now, it 
>>> seems like it might be possible to write the file, start making use 
>>> of it, and then take a power outage before it actually gets written 
>>> to persistent storage.  When we come back up we could have an 
>>> instance expecting to make use of it, but no target information in the on-disk copy of the file.
> 
> I think even if there is no target information in configuration file dir, c-vol started successfully
> and iSCSI targets were created automatically and volumes were exported, right?
> 
> There is an problem in this case that the iSCSI target was created without authentication because
> we can't get previous authentication from the configuration file.
> 
> I'm curious what kind of problem did you met?
>   
>> If its being kept in sync with DB records, and won't self-heal from 
>> this situation, then yes. e.g. if the overall workflow is something 
>> like
> 
> In my understanding, the provider_auth in database has user name and password for iSCSI target. 
> Therefore if we get authentication from DB, I think we can self-heal from this situation
> correctly after c-vol service is restarted.
> 

Is this not already done as-needed by ensure_export()?

> The lio target obtains authentication from provider_auth in database, but tgtd, iet, cxt obtain
> authentication from file to recreate iSCSI target when c-vol is restarted.
> If the file is missing, these volumes are exported without authentication and configuration
> file is recreated as I mentioned above.
> 
> tgtd: Get target chap auth from file
> iet:  Get target chap auth from file
> cxt:  Get target chap auth from file
> lio:  Get target chap auth from Database(in provider_auth)
> scst: Get target chap auth by using original command
> 
> If we get authentication from DB for tgtd, iet and cxt same as lio, we can recreate iSCSI target
> with proper authentication when c-vol is restarted.
> I think this is a solution for this situation.
> 

This may be possible, but fixing the target config file to be written
more safely to work as currently intended is still a win.

> Any thought?
> 
> Thanks,
> Mitsuhiro Tanino
> 
>> -----Original Message-----
>> From: Chris Friesen [mailto:chris.friesen at windriver.com]
>> Sent: Friday, September 25, 2015 12:48 PM
>> To: openstack-dev at lists.openstack.org
>> Subject: Re: [openstack-dev] [cinder] should we use fsync when writing iscsi
>> config file?
>>
>> On 09/24/2015 04:21 PM, Chris Friesen wrote:
>>> On 09/24/2015 12:18 PM, Chris Friesen wrote:
>>>
>>>>
>>>> I think what happened is that we took the SIGTERM after the open()
>>>> call in create_iscsi_target(), but before writing anything to the file.
>>>>
>>>>          f = open(volume_path, 'w+')
>>>>          f.write(volume_conf)
>>>>          f.close()
>>>>
>>>> The 'w+' causes the file to be immediately truncated on opening,
>>>> leading to an empty file.
>>>>
>>>> To work around this, I think we need to do the classic "write to a
>>>> temporary file and then rename it to the desired filename" trick.
>>>> The atomicity of the rename ensures that either the old contents or the new
>> contents are present.
>>>
>>> I'm pretty sure that upstream code is still susceptible to zeroing out
>>> the file in the above scenario.  However, it doesn't take an
>>> exception--that's due to a local change on our part that attempted to fix the
>> below issue.
>>>
>>> The stable/kilo code *does* have a problem in that when it regenerates
>>> the file it's missing the CHAP authentication line (beginning with
>> "incominguser").
>>
>> I've proposed a change at https://urldefense.proofpoint.com/v2/url?u=https-
>> 3A__review.openstack.org_-23_c_227943_&d=BQICAg&c=DZ-
>> EF4pZfxGSU6MfABwx0g&r=klD1krzABGW034E9oBtY1xmIn3g7xZAIxV0XxaZpkJE&m=SVlOqKiqO04_
>> NttKUIoDiaOR7cePB0SOA1bpjakqAss&s=q2_8XBAVH9lQ2mdT72nW7dN2EafIqJEpHGLBuf4K970&e=
>>
>> If anyone has suggestions on how to do this more robustly or more cleanly,
>> please let me know.
>>
>> Chris
>>