Open Stack

Right, I should have mentioned that I thought swift already had these 
capabilities. Sorry! :-)

So to me it seems like server-side processing of artifacts can wait. I 
don't see any reason the workers can't decide at their level how to 
format their logs and therefore who to process them through. Maybe down 
the road we do end up with some middle-man type processor who just 
pushes to swift itself.

Cheers,
Josh

--
Rackspace Australia

On 9/13/13 2:26 AM, James E. Blair wrote:
> Thierry Carrez <thierry at openstack.org> writes:
>
>> Joshua Hesketh wrote:
>>> Great overview and plan James, thanks for that :-).
>>>
>>> So it seems to me that we're duplicating the job of swift a little bit
>>> by writing a program to accept an object over http and store it on disk.
>>> If our end-game is logs stored in swift then why not make jenkins (and
>>> other workers) push the logs straight to swift?
>> I suspect the additional benefit is the token-based security which means
>> the slaves can have limited access to storage (they basically get a
>> token that can only be used for one very limited - and one-time -  task).
> Indeed -- we can't trust the systems producing artifacts (logs,
> tarballs, docs, etc) at all, so we want to tightly scope what they are
> able to publish/store.  For example, a job should only be able to write
> logs to a very specific path determined ahead of time by Zuul or a
> tarball job should be able to publish a tarball only for the project in
> question.
>
>> That said I agree that reinventing HTTP POST storage sounds overkill and
>> storing directly to swift would be a lot simpler. Does Swift include
>> complex access rules that we could leverage to implement the same kind
>> of security that James described in his original post ? If not, would it
>> make sense to actually add what we need to Swift itself ? (after all,
>> it's, you know, an OpenStack project)
> It turns out that there may very well be a mechanism in swift to do what
> we want -- but it's not documented in the swift API documentation!
>
> Compare:
>
>    http://docs.openstack.org/api/openstack-object-storage/1.0/content/
>
> To:
>
>    http://docs.rackspace.com/files/api/v1/cf-devguide/content/
>
> I filed https://bugs.launchpad.net/openstack-manuals/+bug/1224562 for this.
>
> At any rate, the feature that I apparently re-invented is called
> "FormPost", and it's described here:
>
>    http://docs.rackspace.com/files/api/v1/cf-devguide/content/FormPost-d1a555.html
>
> And here (see, it really is part of swift):
>
>    https://git.openstack.org/cgit/openstack/swift/tree/swift/common/middleware/formpost.py
>
> It appears to support everything I specified in my token idea: HMAC
> validation, expiration time, container and sub-path restrictions.  As
> well as max file size and max number of files.
>
> We would lose the ability to do server-side processing of artifacts
> before storage, though of course we still have the opportunity to
> process them on the workers before sending them to swift.
>
> We also need to deal with how to generate indexes -- without the
> artifact-receiving component to add entries to a database, we may either
> need something else to do that, or have something else create directory
> markers in swift so that the artifact-server can create directory
> indexes on the fly.
>
> -Jim
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra