Open Stack

Dear Kota,

On Mon, Sep 17, 2018 at 11:43 PM Kota TSUYUZAKI
<tsuyuzaki.kota at lab.ntt.co.jp> wrote:
>
> Hi Quio,
>
> > I know Storlets can provide user-defined computation functionalities,
> > but I guess some capabilities can only be achieved using middleware.
> > For example, a user may want such a feature: upon each PUT request, it
> > creates a compressed copy of the object and stores both the original
> > copy and compressed copy. It's feasible using middlware but I don't
> > think Storlets provide such capability.
>
> Interesting, exactly currently it's not supported to write to multi objects for a PUT request but as well as other middlewares we could adopt the feasibility into Storlets if you prefer.
> Right now, the multi read (i.e. GET from multi sources) is only available and I think we would be able to expand the logic to PUT requests too. IIRC, in those days, we had discussion on sort of the
> multi-out use cases and I'm sure the data structure inside Storlets are designed to be capable to that expantion. At that time, we called them "Tee" application on Storlets, I could not find the
> historical discussion logs about how to implement tho, sorry. I believe that would be an use case for storlets if you prefer the user-defined application flexibilities rather than operator defined
> Swift middleware.
>
> The example of multi-read (GET from multi sources) are here:
> https://github.com/openstack/storlets/blob/master/tests/functional/python/test_multiinput_storlet.py
>
> And if you like to try to write multi write, please join us, I'm happy to help you anytime.
>

Thanks! I'm interested and would like to join, as well as contribute!

Another potential use case: imagine I want to compress objects upon
PUTs using two different algorithms X and Y, and use the future
'multi-write' feature to store three objects upon any single PUT
(original copy, X-compressed copy and Y-compressed copy). I can
install two Storlets which implement X and Y respectively. However,
seems Storlets engine can only invoke one per PUT, so this is still
not feasible. Is that correct?

>
> > Another example is that a user may want to install a Swift3-like
> > middleware to provide APIs to a 3rd party, but she doesn't want other
> > users to see this middleware.
> >
>
> If the definition can be made by operators, perhaps one possible solution that preparing different proxy-server endpoint for different users is available. i.e. an user uses no-s3api available proxy,
> then the others use a different proxy-server endpoint that has the s3api in the pipeline.
>
> Or, it sounds like kinda defaulter middleware[1], I don't think it has the scope turning on/off the middlewares for now.
>
> 1: https://review.openstack.org/#/c/342857/

I see, thanks for pointing out the defaulter project!

Best,
Qiao

>
> Best,
> Kota
>
> (2018/09/18 11:34), Qiao Kang wrote:
> > Kota,
> >
> > Thanks for your reply, very helpful!
> >
> > I know Storlets can provide user-defined computation functionalities,
> > but I guess some capabilities can only be achieved using middleware.
> > For example, a user may want such a feature: upon each PUT request, it
> > creates a compressed copy of the object and stores both the original
> > copy and compressed copy. It's feasible using middlware but I don't
> > think Storlets provide such capability.
> >
> > Another example is that a user may want to install a Swift3-like
> > middleware to provide APIs to a 3rd party, but she doesn't want other
> > users to see this middleware.
> >
> > Regards,
> > Qiao
> >
> > On Mon, Sep 17, 2018 at 9:19 PM Kota TSUYUZAKI
> > <tsuyuzaki.kota at lab.ntt.co.jp> wrote:
> >>
> >> With Storlets, users will be able to create their own applications that are able to run like as a Swift middeleware. The application (currently Python and Java are supported as the language but the
> >> apps can calls any binaries in the workspace) can be uploaded as a Swift object, then, users can invoke them with just an extra header that specifies your apps.
> >>
> >> To fit your own use case, we may have to consider to invole or to integrate the system for you but I believe Storlets could be a choice for you.
> >>
> >> In detail, Storlets documantation is around there,
> >>
> >> Top Level Index: https://docs.openstack.org/storlets/latest/index.html
> >> System Overview: https://docs.openstack.org/storlets/latest/storlet_engine_overview.html
> >> APIs: https://docs.openstack.org/storlets/latest/api/overview_api.html
> >>
> >> Thanks,
> >>
> >> Kota
> >>
> >> (2018/09/17 8:59), John Dickinson wrote:
> >>> You may be interested in Storlets. It's another OpenStack project, maintained by a Swift core reviewer, that provides this sort of user-defined middleware functionality.
> >>>
> >>> You can also ask about it in #openstack-swift
> >>>
> >>> --John
> >>>
> >>>
> >>>
> >>> On 16 Sep 2018, at 9:25, Qiao Kang wrote:
> >>>
> >>>> Hi,
> >>>>
> >>>> I'm wondering whether Swift allows any user (not the administrator) to
> >>>> specify which middleware that she/he wants his data object to go throught.
> >>>> For instance, Alice wants to install a middleware but doesn't want Bob to
> >>>> use it, where Alice and Bob are two accounts in a single Swift cluster.
> >>>>
> >>>> Or maybe all middlewares are pre-installed globally and cannot be
> >>>> customized on a per-account basis?
> >>>>
> >>>> Thanks,
> >>>> Qiao
> >>>> _______________________________________________
> >>>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >>>> Post to     : openstack at lists.openstack.org
> >>>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >>>
> >>> _______________________________________________
> >>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >>> Post to     : openstack at lists.openstack.org
> >>> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >>
> >>
> >> --
> >> ----------------------------------------------------------
> >> Kota Tsuyuzaki(露﨑　浩太)  <tsuyuzaki.kota at lab.ntt.co.jp>
> >> NTT Software Innovation Center
> >> Distributed Computing Technology Project
> >> Phone  0422-59-2837
> >> Fax    0422-59-2965
> >> -----------------------------------------------------------
> >>
> >>
> >> _______________________________________________
> >> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >> Post to     : openstack at lists.openstack.org
> >> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> >
> >
>
>
> --
> ----------------------------------------------------------
> Kota Tsuyuzaki(露﨑　浩太)  <tsuyuzaki.kota at lab.ntt.co.jp>
> NTT Software Innovation Center
> Distributed Computing Technology Project
> Phone  0422-59-2837
> Fax    0422-59-2965
> -----------------------------------------------------------
>