[Openstack] ownCloud and openstack

Tom Fifield tom at openstack.org
Mon Jul 14 04:36:36 UTC 2014


On 13/07/14 13:48, Remo Mattei wrote:
> As far as I know max size is 5gb into swift unless you setup a way to
> split it

5GB is the default limit for a single request. However, swift provides a 
couple of ways to upload larger single objects:

http://docs.openstack.org/developer/swift/overview_large_objects.html

OwnCloud should probably implement one of these methods to allow objects 
greater than the default limit.

Regards,


Tom


> Il giorno Jul 12, 2014, alle ore 16:42, Jason Bishop
> <jason.bishop at gmail.com <mailto:jason.bishop at gmail.com>> ha scritto:
>
>>
>> Hi Frank, I have to say ownCloud is really nicely done.  i have been
>> in need of a file share which has native linux client support and a
>> really robust storage backend.  ownCloud with swift meets the
>> requirements.  I do have a comment which mirrors Hugo's comments I
>> think.  Is it feasible to have the concept of users in ownCloud more
>> closely match up with swift's concept?  It would be nice if the
>> backend storage for each user was stored in that user's swift account.
>>  I'm sure it will add complexity to user management and authentication
>> but the advantages of separation of user data may offset that.
>>
>> I did some testing on ubuntu trusty against an ubuntu trusty icehouse
>> openstack.  I updated the config file and didn't get it quite right
>> the first time.  The error in the admin gui told me what the problem
>> was (I had wrong string in for the region attribute) so that worked nice.
>>
>> I stood up an instance of trusty and installed the owncloud client and
>> copied some files into the shared directory.
>>
>> a 2gb file worked fine.  when i tried a 7gb file I got this error
>> message, so I think the code in ownCloud does not handle > 5GB files
>> in swift.  It also looks like it keeps retrying the upload over and
>> over.  i'll watch it to see if it eventually gives up and errors out.
>>
>> Fatal webdav Exception: #9 {main} 2014-07-12T16:11:43+00:00
>> Fatal webdav Exception: #8 /var/www/html/owncloud/remote.php(52):
>> require_once('/var/www/html/o...') 2014-07-12T16:11:43+00:00
>> Fatal webdav Exception: #7
>> /var/www/html/owncloud/apps/files/appinfo/remote.php(61):
>> Sabre\DAV\Server->exec() 2014-07-12T16:11:43+00:00
>> FatalwebdavException: #6
>> /var/www/html/owncloud/3rdparty/sabre/dav/lib/Sabre/DAV/Server.php(214):
>> Sabre\DAV\Server->invokeMethod('PUT',
>> '7gigafile-chunk...')2014-07-12T16:11:43+00:00
>> FatalwebdavException: #5
>> /var/www/html/owncloud/3rdparty/sabre/dav/lib/Sabre/DAV/Server.php(474):
>> call_user_func(Array, '7gigafile-chunk...')2014-07-12T16:11:43+00:00
>> FatalwebdavException: #4 [internal function]:
>> Sabre\DAV\Server->httpPut('7gigafile-chunk...')2014-07-12T16:11:43+00:00
>> FatalwebdavException: #3
>> /var/www/html/owncloud/3rdparty/sabre/dav/lib/Sabre/DAV/Server.php(900):
>> Sabre\DAV\Server->createFile('7gigafile-chunk...', Resource id #17,
>> NULL)2014-07-12T16:11:43+00:00
>> FatalwebdavException: #2
>> /var/www/html/owncloud/3rdparty/sabre/dav/lib/Sabre/DAV/Server.php(1647):
>> OC_Connector_Sabre_Directory->createFile('7gigafile-chunk...',
>> Resource id #17)2014-07-12T16:11:43+00:00
>> FatalwebdavException: #1
>> /var/www/html/owncloud/lib/private/connector/sabre/directory.php(76):
>> OC_Connector_Sabre_File->put(Resource id #17)2014-07-12T16:11:43+00:00
>> FatalwebdavSabre\DAV\Exception: Could not rename part file assembled
>> from chunks2014-07-12T16:11:43+00:00
>> Errorwebdav\OC\Files\Filesystem::rename() failed2014-07-12T16:11:43+00:00
>> ErrorobjectstoreCould not create object: Client error response [status
>> code] 413 [reason phrase] Request Entity Too Large [url]
>> http://openstack-swiftproxy.stanford.edu:8080/v1/AUTH_78043e536430433fae0318b0ac1040ff/owncloud2/urn:oid:1163
>>
>> I will continue testing and let you know if I find anything else.  I
>> am still using just http connection for keystone and swift services.
>>  I will pull proper https certificates and check ssl plumbing is
>> working fine.
>>
>> After that my next steps will probably be authentication and LDAP related.
>>
>> Jason
>>
>>
>>
>> On Mon, Jul 7, 2014 at 9:09 AM, Frank Karlitschek <frank at owncloud.com
>> <mailto:frank at owncloud.com>> wrote:
>>
>>     Hi Hugo,
>>
>>
>>     thanks a lot for the recommendations. This is super helpful.
>>     We will look into the upload problem you found.
>>
>>
>>     Cheers
>>     Frank
>>
>>
>>     > On 07.07.2014, at 05:09, Kuo Hugo <tonytkdk at gmail.com
>>     <mailto:tonytkdk at gmail.com>> wrote:
>>     >
>>     > Hi Frank,
>>     >
>>     > I had a try with it on my box here. Most of OwnCloud's features
>>     are done by OwnCloud host. Not too much functions rely on Swift's
>>     feature. It simply uses CRUD methods.
>>     >
>>     > [Suggestions]
>>     >       • Multiple containers/accounts for Swift as primary
>>     storage. The performance will degrade while over 1million within a
>>     container.
>>     >       • Since all metadata is stored in the DB on OwnCloud host.
>>     And OwnCloud remaps the filename as object_id in Swift. That would
>>     be good to store some mapping metadata in Swift for safety. If the
>>     DB crashed, no chance to get the proper data from Swift. User has
>>     no knowledge about object name "urn:oid:153".
>>     >       • For large object(100MB+), OwnCloud chunks it into pieces
>>     and upload each segment to Swift as single object. Swift provide
>>     two methods for handling large object DLO/SLO. It might be good to
>>     leverage it for reduce some loading on OwnCloud. And more
>>     consistency of how large object been stored in Swift.
>>     >       • Data traffic passthrough the OwnCloud server will cause
>>     the bottleneck/loading on OwnCloud host. To upload data to Swift
>>     API from client app directly may bring you some benefits.
>>     >       • OwnCloud can have the shared link from Swift directly
>>     via TempURL feature.
>>     >       • The Object expired feature of Swift is a plus for
>>     OwnCloud to use.
>>     >       • Pre-create folders for user in objectstore as primary
>>     storage mode.
>>     >       • When downloading file, seems OwnCloud will cache it
>>     somewhere. Once the entire object were loaded into OwnCloud then
>>     sending to user. To streaming it is an ideally way to do so from
>>     my perspective.
>>     >
>>     > Renaming is easy on OwnCloud by overwriting the value in DB.
>>     > Thumbnail is good idea.
>>     > To edit txt file on Web is awesome.
>>     >
>>     > I got a problem to upload a 470MB object from OSX client app. It
>>     just showed me timeout but no any further clues here.
>>     >
>>     > Cheers // Hugo
>>     >
>>     >
>>     >
>>     >
>>     > 2014-07-04 10:31 GMT+08:00 John Dickinson <me at not.mn
>>     <mailto:me at not.mn>>:
>>     > That's fantastic news! Hearing more companies and applications
>>     supporting Swift is great for the community, and it gives us as
>>     contributors yet another use case to reference.
>>     >
>>     > I work with Hugo, so I'll peek over his shoulder to check it
>>     out. Meanwhile, if there's anything I can do to help answer
>>     questions or get involved in the community (from contributing to
>>     getting issues addressed), please let me know.
>>     >
>>     > --John
>>     >
>>     >
>>     > > On Jul 3, 2014, at 6:01 PM, Frank Karlitschek
>>     <frank at owncloud.com <mailto:frank at owncloud.com>> wrote:
>>     > >
>>     > > Hi everyone,
>>     > >
>>     > >
>>     > > Tom Fifield recommended that I send a message to this list.
>>     > >
>>     > > As some of you might know we at ownCloud just added Swift
>>     support as primary storage to ownCloud. This will be released as
>>     part of our upcoming release 7 in about 3 weeks.
>>     > >
>>     > > The code can be seen here.
>>     > > https://github.com/owncloud/core/pull/8383
>>     > >
>>     > >
>>     > > ownCloud is a fully open source solution that can be used a
>>     self hosted Dropbox or Google Drive alternative. The new Swift
>>     backend is an important step for us to mask ownCloud more
>>     scalable, easier to deploy in bigger setups and integrate it
>>     better into OpenStack.
>>     > >
>>     > > Tom suggested to ask here for testers who are willing to try
>>     ownCloud 7 together with Swift.
>>     > > We just announced the RC1 a few minutes ago:
>>     > >
>>     http://mailman.owncloud.org/pipermail/announcements/2014-July/000049.html
>>     > >
>>     > >
>>     > > So it would be great if you guys could give it a test and give
>>     some feedback.
>>     > >
>>     > >
>>     > > Thanks a lot
>>     > >
>>     > >
>>     > > Cheers
>>     > > Frank
>>     > > _______________________________________________
>>     > > Mailing list:
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     > > Post to     : openstack at lists.openstack.org
>>     <mailto:openstack at lists.openstack.org>
>>     > > Unsubscribe :
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     >
>>     > _______________________________________________
>>     > Mailing list:
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     > Post to     : openstack at lists.openstack.org
>>     <mailto:openstack at lists.openstack.org>
>>     > Unsubscribe :
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     >
>>
>>
>>     _______________________________________________
>>     Mailing list:
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>     Post to     : openstack at lists.openstack.org
>>     <mailto:openstack at lists.openstack.org>
>>     Unsubscribe :
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>> !DSPAM:1,53c1cac3265432135880153!
>> _______________________________________________
>> Mailing list:
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> <mailto:openstack at lists.openstack.org>
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>> !DSPAM:1,53c1cac3265432135880153!
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>





More information about the Openstack mailing list