Open Stack

Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > >
> > >After having some conversations with folks at the Ops Midcycle a
> > >few weeks ago, and observing some of the more recent email threads
> > >related to glance, glance-store, the client, and the API, I spent
> > >last week contacting a few of you individually to learn more about
> > >some of the issues confronting the Glance team. I had some very
> > >frank, but I think constructive, conversations with all of you about
> > >the issues as you see them. As promised, this is the public email
> > >thread to discuss what I found, and to see if we can agree on what
> > >the Glance team should be focusing on going into the Mitaka summit
> > >and development cycle and how the rest of the community can support
> > >you in those efforts.
> > >
> > >I apologize for the length of this email, but there's a lot to go
> > >over. I've identified 2 high priority items that I think are critical
> > >for the team to be focusing on starting right away in order to use
> > >the upcoming summit time effectively. I will also describe several
> > >other issues that need to be addressed but that are less immediately
> > >critical. First the high priority items:
> > >
> > >1. Resolve the situation preventing the DefCore committee from
> > >   including image upload capabilities in the tests used for trademark
> > >   and interoperability validation.
> > >
> > >2. Follow through on the original commitment of the project to
> > >   provide an image API by completing the integration work with
> > >   nova and cinder to ensure V2 API adoption.
> > 
> > Hi Doug,
> > 
> > First and foremost, I'd like to thank you for taking the time to dig
> > into these issues, and for reaching out to the community seeking for
> > information and a better understanding of what the real issues are. I
> > can imagine how much time you had to dedicate on this and I'm glad you
> > did.
> > 
> > Now, to your email, I very much agree with the priorities you
> > mentioned above and I'd like for, whomever will win Glance's PTL
> > election, to bring focus back on that.
> > 
> > Please, find some comments in-line for each point:
> > 
> > >
> > >I. DefCore
> > >
> > >The primary issue that attracted my attention was the fact that
> > >DefCore cannot currently include an image upload API in its
> > >interoperability test suite, and therefore we do not have a way to
> > >ensure interoperability between clouds for users or for trademark
> > >use. The DefCore process has been long, and at times confusing,
> > >even to those of us following it sort of closely. It's not entirely
> > >surprising that some projects haven't been following the whole time,
> > >or aren't aware of exactly what the whole thing means. I have
> > >proposed a cross-project summit session for the Mitaka summit to
> > >address this need for communication more broadly, but I'll try to
> > >summarize a bit here.
> > 
> > +1
> > 
> > I think it's quite sad that some projects, especially those considered
> > to be part of the `starter-kit:compute`[0], don't follow closely
> > what's going on in DefCore. I personally consider this a task PTLs
> > should incorporate in their role duties. I'm glad you proposed such
> > session, I hope it'll help raising awareness of this effort and it'll
> > help moving things forward on that front.
> 
> Until fairly recently a lot of the discussion was around process
> and priorities for the DefCore committee. Now that those things are
> settled, and we have some approved policies, it's time to engage
> more fully.  I'll be working during Mitaka to improve the two-way
> communication.
> 
> > 
> > >
> > >DefCore is using automated tests, combined with business policies,
> > >to build a set of criteria for allowing trademark use. One of the
> > >goals of that process is to ensure that all OpenStack deployments
> > >are interoperable, so that users who write programs that talk to
> > >one cloud can use the same program with another cloud easily. This
> > >is a *REST API* level of compatibility. We cannot insert cloud-specific
> > >behavior into our client libraries, because not all cloud consumers
> > >will use those libraries to talk to the services. Similarly, we
> > >can't put the logic in the test suite, because that defeats the
> > >entire purpose of making the APIs interoperable. For this level of
> > >compatibility to work, we need well-defined APIs, with a long support
> > >period, that work the same no matter how the cloud is deployed. We
> > >need the entire community to support this effort. From what I can
> > >tell, that is going to require some changes to the current Glance
> > >API to meet the requirements. I'll list those requirements, and I
> > >hope we can discuss them to a degree that ensures everyone understands
> > >them. I don't want this email thread to get bogged down in
> > >implementation details or API designs, though, so let's try to keep
> > >the discussion at a somewhat high level, and leave the details for
> > >specs and summit discussions. I do hope you will correct any
> > >misunderstandings or misconceptions, because unwinding this as an
> > >outside observer has been quite a challenge and it's likely I have
> > >some details wrong.
> > >
> > >As I understand it, there are basically two ways to upload an image
> > >to glance using the V2 API today. The "POST" API pushes the image's
> > >bits through the Glance API server, and the "task" API instructs
> > >Glance to download the image separately in the background. At one
> > >point apparently there was a bug that caused the results of the two
> > >different paths to be incompatible, but I believe that is now fixed.
> > >However, the two separate APIs each have different issues that make
> > >them unsuitable for DefCore.
> > >
> > >The DefCore process relies on several factors when designating APIs
> > >for compliance. One factor is the technical direction, as communicated
> > >by the contributor community -- that's where we tell them things
> > >like "we plan to deprecate the Glance V1 API". In addition to the
> > >technical direction, DefCore looks at the deployment history of an
> > >API. They do not want to require deploying an API if it is not seen
> > >as widely usable, and they look for some level of existing adoption
> > >by cloud providers and distributors as an indication of that the
> > >API is desired and can be successfully used. Because we have multiple
> > >upload APIs, the message we're sending on technical direction is
> > >weak right now, and so they have focused on deployment considerations
> > >to resolve the question.
> > 
> > The task upload process you're referring to is the one that uses the
> > `import` task, which allows you to download an image from an external
> > source, asynchronously, and import it in Glance. This is the old
> > `copy-from` behavior that was moved into a task.
> > 
> > The "fun" thing about this - and I'm sure other folks in the Glance
> > community will disagree - is that I don't consider tasks to be a
> > public API. That is to say, I would expect tasks to be an internal API
> > used by cloud admins to perform some actions (bsaed on its current
> > implementation). Eventually, some of these tasks could be triggered
> > from the external API but as background operations that are triggered
> > by the well-known public ones and not through the task API.
> 
> Does that mean it's more of an "admin" API?
> 

I think it is basically just a half-way done implementation that is
exposed directly to users of Rackspace Cloud and, AFAIK, nobody else.
When last I tried to make integration tests in shade that exercised the
upstream glance task import code, I was met with an implementation that
simply did not work, because the pieces behind it had never been fully
implemented upstream. That may have been resolved, but in the process
of trying to write tests and make this work, I discovered a system that
made very little sense from a user standpoint. I want to upload an
image, why do I want a task?!

> > 
> > Ultimately, I believe end-users of the cloud simply shouldn't care
> > about what tasks are or aren't and more importantly, as you mentioned
> > later in the email, tasks make clouds not interoperable. I'd be pissed
> > if my public image service would ask me to learn about tasks to be
> > able to use the service.
> 
> It would be OK if a public API set up to do a specific task returned a
> task ID that could be used with a generic task API to check status, etc.
> So the idea of tasks isn't completely bad, it's just too vague as it's
> exposed right now.
> 

I think it is a concern, because it is assuming users will want to do
generic things with a specific API. This turns into a black-box game where
the user shoves a task in and then waits to see what comes out the other
side. Not something I want to encourage users to do or burden them with.

We have an API whose sole purpose is to accept image uploads. That
Rackspace identified a scaling pain point there is _good_. But why not
*solve* it for the user, instead of introduce more complexity?

What I'd like to see is the upload image API given the ability to
respond with a URL that can be uploaded to using the object storage API
we already have in OpenStack. Exposing users to all of these operator
choices is just wasting their time. Just simply say "Oh, you want to
upload an image? Thats fine, please upload it as an object over there
and POST here again when it is ready to be imported." This will make
perfect sense to a user reading docs, and doesn't require them to grasp
an abstract concept like "tasks" when all they want to do is upload
their image.