Open Stack

Hi folks,

I’ve noticed that the Community Application Catalog has begun to implement
its own API, and it seems to me that we are going to have some significant
duplication of efforts with the code which has already been implemented in
Glance as Artifact Repository initiative (also known as Glance V3).
>From the very beginning of the App Catalog project (and I’ve been involved
in it since February) I’ve been proposing to use Glance as its backend,
because from my point of view it covers like 90% of the needed
functionality. But it looks like we have some kind of miscommunication
here, as I am getting some confusing questions and assumptions, like the
vision of Glance V3 being dedicated backend for Murano (which is definitely
incorrect).
So, I am writing the email to clarify my vision of what Glance V3 is and
how its features may be used to provide the REST API for Community App
Catalog.

1.  Versioned schema
First of all, Glance V3 operates on entities called “artifacts”, and these
ones perfectly map to the Data Assets of community app catalog. The
artifacts are strongly typed: there are artifact types for murano packages,
glance images, heat templates - and there may be (and will be) more. Each
artifact type is represented by a plugin, defining the schema of objects’
data and metadata and - optionally - custom logic. So, this thing is
extensible: when a new type of asset needs to be added to a catalog it can
be done really quickly by just defining the schema and putting that schema
into a plugin. Also, these plugins are versioned, so the possible changes
in the schema are handled properly.

2. Generic properties
Next, all the artifact types in Glance V3 have some generic metadata
properties (i.e. part of the schema which is common for all the types),
including the name, the version, description, authorship information and so
on. This also corresponds to the data schema of community app catalog. The
mapping is not 1:1, but we can work together on this to make sure that
these generic properties match the expectations of the catalog.

3. Versioning
Versions are very important for catalogs of objects, so Glance V3 was
initially designed keeping versioning questions in mind: each artifact has
a semver-based version assigned, so the artifacts having the same name but
different versions are grouped into the proper sequences. API is able to
query artifacts based on their version spec, e.g. it is possible to fetch
the latest artifact with the name “foo” having the version greater than 2.1
and less than 3.5.7 - or any other version spec, similar to pip or any
other similar tool. As far as I know, community app catalog does not have
such capabilities right now - and I strongly believe that it is really a
must have feature for a catalog to be successful. At least it is absolutely
mandatory for Murano packages, which are the only “real apps” among the
asset types right now.

4. Cross artifact dependencies
Glance V3 also has the dependency relations from the very beginning, so
they may be defined as part of artifact type schema. As a result, an
artifact may “reference” any number of other artifacts with various
semantic. For example, murano package may define a set of references to
other murano packages and call it “requires” - and this will act similar to
the requirements of a python package. Similar properties may be defined for
heat templates and glance images - they may reference each other with
various semantics.
Of course, the definitions of such dependencies may be done internally
inside the packages, so they may be resolved locally by the service which
is going to use it, but letting the catalog know about them will allow us
to do the import-export operations for any given artifacts and its
dependencies automatically, only by the means of the catalog itself.

5. Search and filtering API
Right now Glance V3 API is in experimental state (we plan to stabilize it
during the Mitaka cycle), but it already provides quite good capabilities
to discover things. It can search artifacts by their type, name and
(optionally) aforementioned version specs, by tag or even by arbitrary set
of metadata properties. We have plans to integrate Glance V3 with the
Searchlight project to have even more index and search capabilities using
its elastic search engine.

6. Data storage
As you probably know, Glance does not own the binary data of its images.
Instead, it provides an abstraction of the backend storage, which may be
swift, ceph, s3 or something else. The same approach is used in Glance V3
for artifacts data, but with more per-type control: particular artifact
types may be configured independently to store their blobs in different
backends. This may be of use for Community App Catalog which operates on
different storages for its assets.

7. Sharing and access control.
Glance V3 inherits the same access mechanics present in Glance V2: an
artifact may be visible to its owner tenant only, be public (i.e. visible
to all the tenants) or directly shared by the owner to a specific tenant.
Also, Glance can act in the anonymous mode (i.e. without an access token),
thus providing access to public artifacts even to unauthenticated users.
This can be easily applied to a public web service, such as community app
catalog: regular unauthenticated users use anonymous mode to access only
the public assets (this is the current behavior of apps.o.o), while
registered users will have their own private spaces (“tenants”) with
various access restrictions.

8. The federation.
The ultimate goal for Glance Artifact Repository is ability to build trees
of artifact repos in different clouds. The top node of that tree is some
Global Application Catalog (and the apps.openstack.org may be this global
catalog - if it is glance-based or at least supports glance v3 federation),
then there are repositories of particular openstack vendors or
distributors, then - the catalogs of enterprises operating different
openstack clouds. The particular glance deployments in that clouds are the
leafs of that tree, being able to search for data assets in all the
upstream repositories, download them from there or - if permitted - submit
their local assets back upstream. This will be the ultimate network for
application delivery and exchange in openstack world - and this is one of
the main reasons we’ve began the Artifacts initiative in Glance.
Unlike other aforementioned features this one is not implemented yet, but
we are planning to add it as soon as we are done with API stabilization
goal.


There are many other features which are present in V3’s roadmap and may be
useful for the app catalog, such as ability to sign artifacts with their
developers’ keys and verify that keys on usage to ensure the authenticity
of the artifact.

What we don’t have right now is the ability to associate ratings (“stars”)
and comments to the artifact, as well as aggregating different usage and
download statistics: such features are really needed only for the public
website such as apps.o.o but are not required for Glance’s in particular
clouds. But we may find some way to solve this, either by wrapping glance
API with additional middleware which would add appropriate info from a
different data source, or by having custom plugins which are able to do
that, or in some other way: I am sure we may find a solution for this.

So, this was just a brief description of what Glance v3 has to offer as a
backend for App Catalog API.
It also worths to mention that this API is in “EXPERIMENTAL” state right
now, which means that it is not fixed and we may modify it significantly if
there is a need to. So we may work closer together to adopt it for the
needs of Community App Catalog.

I would really prefer to not create any overlaps between Glance v3 and the
community app catalog: if the app catalog builds its own incompatible
implementation of assets discovery and distribution API then we’ll have a
huge duplication of efforts for developers and lots of confusion to the
end-users who will get two entirely different ways to do the same task.

So, I’d propose to discuss these potential overlaps, look at the features
need by App Catalog and see how Glance V3 may be of use here. I’ll be more
than happy to help with that. We can dive deeper into the details here in
the mailing list or meet in person in Tokyo. I'll try to have a
demonstratable prototype by that time.

--
Regards,
Alexander Tivelkov
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151015/64165f50/attachment-0001.html>