Open Stack

Hi folks!

Thanks all for the feedback. Yeah, this definitely needs more discussions,
and the Summit is good place to have them. Thanks to Serg for offering a
spot in the schedule.
I hope to have some PoC by that time, so we can have something which can be
demoed.

I'll try to attend the AppCatalog sessions as well. I didn't finalize my
summit schedule yet, but I'll do my best to attend as much as I can.

Thanks!


--
Regards,
Alexander Tivelkov

On Fri, Oct 16, 2015 at 10:07 PM, Christopher Aedo <doc at aedo.net> wrote:

> On Thu, Oct 15, 2015 at 3:04 AM, Alexander Tivelkov
> <ativelkov at mirantis.com> wrote:
> > Hi folks,
> >
> > I’ve noticed that the Community Application Catalog has begun to
> implement
> > its own API, and it seems to me that we are going to have some
> significant
> > duplication of efforts with the code which has already been implemented
> in
> > Glance as Artifact Repository initiative (also known as Glance V3).
> > From the very beginning of the App Catalog project (and I’ve been
> involved
> > in it since February) I’ve been proposing to use Glance as its backend,
> > because from my point of view it covers like 90% of the needed
> > functionality. But it looks like we have some kind of miscommunication
> here,
> > as I am getting some confusing questions and assumptions, like the
> vision of
> > Glance V3 being dedicated backend for Murano (which is definitely
> > incorrect).
> > So, I am writing the email to clarify my vision of what Glance V3 is and
> how
> > its features may be used to provide the REST API for Community App
> Catalog.
> >
> > 1.  Versioned schema
> > First of all, Glance V3 operates on entities called “artifacts”, and
> these
> > ones perfectly map to the Data Assets of community app catalog. The
> > artifacts are strongly typed: there are artifact types for murano
> packages,
> > glance images, heat templates - and there may be (and will be) more. Each
> > artifact type is represented by a plugin, defining the schema of objects’
> > data and metadata and - optionally - custom logic. So, this thing is
> > extensible: when a new type of asset needs to be added to a catalog it
> can
> > be done really quickly by just defining the schema and putting that
> schema
> > into a plugin. Also, these plugins are versioned, so the possible
> changes in
> > the schema are handled properly.
> >
> > 2. Generic properties
> > Next, all the artifact types in Glance V3 have some generic metadata
> > properties (i.e. part of the schema which is common for all the types),
> > including the name, the version, description, authorship information and
> so
> > on. This also corresponds to the data schema of community app catalog.
> The
> > mapping is not 1:1, but we can work together on this to make sure that
> these
> > generic properties match the expectations of the catalog.
> >
> > 3. Versioning
> > Versions are very important for catalogs of objects, so Glance V3 was
> > initially designed keeping versioning questions in mind: each artifact
> has a
> > semver-based version assigned, so the artifacts having the same name but
> > different versions are grouped into the proper sequences. API is able to
> > query artifacts based on their version spec, e.g. it is possible to fetch
> > the latest artifact with the name “foo” having the version greater than
> 2.1
> > and less than 3.5.7 - or any other version spec, similar to pip or any
> other
> > similar tool. As far as I know, community app catalog does not have such
> > capabilities right now - and I strongly believe that it is really a must
> > have feature for a catalog to be successful. At least it is absolutely
> > mandatory for Murano packages, which are the only “real apps” among the
> > asset types right now.
> >
> > 4. Cross artifact dependencies
> > Glance V3 also has the dependency relations from the very beginning, so
> they
> > may be defined as part of artifact type schema. As a result, an artifact
> may
> > “reference” any number of other artifacts with various semantic. For
> > example, murano package may define a set of references to other murano
> > packages and call it “requires” - and this will act similar to the
> > requirements of a python package. Similar properties may be defined for
> heat
> > templates and glance images - they may reference each other with various
> > semantics.
> > Of course, the definitions of such dependencies may be done internally
> > inside the packages, so they may be resolved locally by the service
> which is
> > going to use it, but letting the catalog know about them will allow us
> to do
> > the import-export operations for any given artifacts and its dependencies
> > automatically, only by the means of the catalog itself.
> >
> > 5. Search and filtering API
> > Right now Glance V3 API is in experimental state (we plan to stabilize it
> > during the Mitaka cycle), but it already provides quite good
> capabilities to
> > discover things. It can search artifacts by their type, name and
> > (optionally) aforementioned version specs, by tag or even by arbitrary
> set
> > of metadata properties. We have plans to integrate Glance V3 with the
> > Searchlight project to have even more index and search capabilities using
> > its elastic search engine.
> >
> > 6. Data storage
> > As you probably know, Glance does not own the binary data of its images.
> > Instead, it provides an abstraction of the backend storage, which may be
> > swift, ceph, s3 or something else. The same approach is used in Glance V3
> > for artifacts data, but with more per-type control: particular artifact
> > types may be configured independently to store their blobs in different
> > backends. This may be of use for Community App Catalog which operates on
> > different storages for its assets.
> >
> > 7. Sharing and access control.
> > Glance V3 inherits the same access mechanics present in Glance V2: an
> > artifact may be visible to its owner tenant only, be public (i.e.
> visible to
> > all the tenants) or directly shared by the owner to a specific tenant.
> Also,
> > Glance can act in the anonymous mode (i.e. without an access token), thus
> > providing access to public artifacts even to unauthenticated users.
> > This can be easily applied to a public web service, such as community app
> > catalog: regular unauthenticated users use anonymous mode to access only
> the
> > public assets (this is the current behavior of apps.o.o), while
> registered
> > users will have their own private spaces (“tenants”) with various access
> > restrictions.
> >
> > 8. The federation.
> > The ultimate goal for Glance Artifact Repository is ability to build
> trees
> > of artifact repos in different clouds. The top node of that tree is some
> > Global Application Catalog (and the apps.openstack.org may be this
> global
> > catalog - if it is glance-based or at least supports glance v3
> federation),
> > then there are repositories of particular openstack vendors or
> distributors,
> > then - the catalogs of enterprises operating different openstack clouds.
> The
> > particular glance deployments in that clouds are the leafs of that tree,
> > being able to search for data assets in all the upstream repositories,
> > download them from there or - if permitted - submit their local assets
> back
> > upstream. This will be the ultimate network for application delivery and
> > exchange in openstack world - and this is one of the main reasons we’ve
> > began the Artifacts initiative in Glance.
> > Unlike other aforementioned features this one is not implemented yet,
> but we
> > are planning to add it as soon as we are done with API stabilization
> goal.
> >
> >
> > There are many other features which are present in V3’s roadmap and may
> be
> > useful for the app catalog, such as ability to sign artifacts with their
> > developers’ keys and verify that keys on usage to ensure the
> authenticity of
> > the artifact.
> >
> > What we don’t have right now is the ability to associate ratings
> (“stars”)
> > and comments to the artifact, as well as aggregating different usage and
> > download statistics: such features are really needed only for the public
> > website such as apps.o.o but are not required for Glance’s in particular
> > clouds. But we may find some way to solve this, either by wrapping glance
> > API with additional middleware which would add appropriate info from a
> > different data source, or by having custom plugins which are able to do
> > that, or in some other way: I am sure we may find a solution for this.
> >
> > So, this was just a brief description of what Glance v3 has to offer as a
> > backend for App Catalog API.
> > It also worths to mention that this API is in “EXPERIMENTAL” state right
> > now, which means that it is not fixed and we may modify it significantly
> if
> > there is a need to. So we may work closer together to adopt it for the
> needs
> > of Community App Catalog.
> >
> > I would really prefer to not create any overlaps between Glance v3 and
> the
> > community app catalog: if the app catalog builds its own incompatible
> > implementation of assets discovery and distribution API then we’ll have a
> > huge duplication of efforts for developers and lots of confusion to the
> > end-users who will get two entirely different ways to do the same task.
> >
> > So, I’d propose to discuss these potential overlaps, look at the features
> > need by App Catalog and see how Glance V3 may be of use here. I’ll be
> more
> > than happy to help with that. We can dive deeper into the details here in
> > the mailing list or meet in person in Tokyo. I'll try to have a
> > demonstratable prototype by that time.
>
> Alexander, you make some great points here about the feature overlaps
> between the Glance v3 API and where we are heading with the App
> Catalog.  There's good potential to re-use some of this work for sure,
> and I'm really looking forward to what comes of the PoC you talked
> about.
>
> Two important considerations against adopting Glance v3 wholesale
> (aside from the experimental nature and the possibility that key bits
> won't be ultimately adopted in the finished stable product) are
> development velocity and deployment goals.  As glance is a key
> component in any OpenStack cloud, and is expected to be deployed in
> hundreds (or thousands!) of environments, the dependencies and default
> requirements are tightly coupled to most other OpenStack projects.
> Obviously any changes to glance are weighed against how those will
> impact all these clouds, how upgrades will work, backwards and
> forwards compatibility, etc.  For all the benefits that come with
> working on a distributed open source project like this, it also tends
> to mean we can't always move very fast because of the potential impact
> across the landscape of the whole project.
>
> The App Catalog is in a different position - to put it really simply,
> we're developing a web site intended to benefit user of OpenStack
> clouds, and including some integration points to make it much easier
> for users to find and share things that run on OpenStack clouds.  But
> we're not tightly coupled to any of the other projects, and we're not
> expecting this to be something that's deployed on it's own with every
> OpenStack cloud.  In fact I'd argue deploying a local/private App
> Catalog dilutes the value and harms the ecosystem as a whole.
>
> Ideally we'll find a balanced way to benefit from the great work
> coming from the Glance team without limiting our design considerations
> or encumbering the App Catalog with requirements or limitations that
> shouldn't impact us.
>
> I'm really looking forward to speaking more about this in person in
> Tokyo, hopefully you can make it to the App Catalog sessions!
>
> -Christopher
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151017/652c2d65/attachment.html>