Open Stack

Excerpts from Clint Byrum's message of 2015-09-14 13:25:43 -0700:
> Excerpts from Doug Hellmann's message of 2015-09-14 12:51:24 -0700:
> > Excerpts from Flavio Percoco's message of 2015-09-14 14:41:00 +0200:
> > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
> > > >
> > > >After having some conversations with folks at the Ops Midcycle a
> > > >few weeks ago, and observing some of the more recent email threads
> > > >related to glance, glance-store, the client, and the API, I spent
> > > >last week contacting a few of you individually to learn more about
> > > >some of the issues confronting the Glance team. I had some very
> > > >frank, but I think constructive, conversations with all of you about
> > > >the issues as you see them. As promised, this is the public email
> > > >thread to discuss what I found, and to see if we can agree on what
> > > >the Glance team should be focusing on going into the Mitaka summit
> > > >and development cycle and how the rest of the community can support
> > > >you in those efforts.
> > > >
> > > >I apologize for the length of this email, but there's a lot to go
> > > >over. I've identified 2 high priority items that I think are critical
> > > >for the team to be focusing on starting right away in order to use
> > > >the upcoming summit time effectively. I will also describe several
> > > >other issues that need to be addressed but that are less immediately
> > > >critical. First the high priority items:
> > > >
> > > >1. Resolve the situation preventing the DefCore committee from
> > > >   including image upload capabilities in the tests used for trademark
> > > >   and interoperability validation.
> > > >
> > > >2. Follow through on the original commitment of the project to
> > > >   provide an image API by completing the integration work with
> > > >   nova and cinder to ensure V2 API adoption.
> > > 
> > > Hi Doug,
> > > 
> > > First and foremost, I'd like to thank you for taking the time to dig
> > > into these issues, and for reaching out to the community seeking for
> > > information and a better understanding of what the real issues are. I
> > > can imagine how much time you had to dedicate on this and I'm glad you
> > > did.
> > > 
> > > Now, to your email, I very much agree with the priorities you
> > > mentioned above and I'd like for, whomever will win Glance's PTL
> > > election, to bring focus back on that.
> > > 
> > > Please, find some comments in-line for each point:
> > > 
> > > >
> > > >I. DefCore
> > > >
> > > >The primary issue that attracted my attention was the fact that
> > > >DefCore cannot currently include an image upload API in its
> > > >interoperability test suite, and therefore we do not have a way to
> > > >ensure interoperability between clouds for users or for trademark
> > > >use. The DefCore process has been long, and at times confusing,
> > > >even to those of us following it sort of closely. It's not entirely
> > > >surprising that some projects haven't been following the whole time,
> > > >or aren't aware of exactly what the whole thing means. I have
> > > >proposed a cross-project summit session for the Mitaka summit to
> > > >address this need for communication more broadly, but I'll try to
> > > >summarize a bit here.
> > > 
> > > +1
> > > 
> > > I think it's quite sad that some projects, especially those considered
> > > to be part of the `starter-kit:compute`[0], don't follow closely
> > > what's going on in DefCore. I personally consider this a task PTLs
> > > should incorporate in their role duties. I'm glad you proposed such
> > > session, I hope it'll help raising awareness of this effort and it'll
> > > help moving things forward on that front.
> > 
> > Until fairly recently a lot of the discussion was around process
> > and priorities for the DefCore committee. Now that those things are
> > settled, and we have some approved policies, it's time to engage
> > more fully.  I'll be working during Mitaka to improve the two-way
> > communication.
> > 
> > > 
> > > >
> > > >DefCore is using automated tests, combined with business policies,
> > > >to build a set of criteria for allowing trademark use. One of the
> > > >goals of that process is to ensure that all OpenStack deployments
> > > >are interoperable, so that users who write programs that talk to
> > > >one cloud can use the same program with another cloud easily. This
> > > >is a *REST API* level of compatibility. We cannot insert cloud-specific
> > > >behavior into our client libraries, because not all cloud consumers
> > > >will use those libraries to talk to the services. Similarly, we
> > > >can't put the logic in the test suite, because that defeats the
> > > >entire purpose of making the APIs interoperable. For this level of
> > > >compatibility to work, we need well-defined APIs, with a long support
> > > >period, that work the same no matter how the cloud is deployed. We
> > > >need the entire community to support this effort. From what I can
> > > >tell, that is going to require some changes to the current Glance
> > > >API to meet the requirements. I'll list those requirements, and I
> > > >hope we can discuss them to a degree that ensures everyone understands
> > > >them. I don't want this email thread to get bogged down in
> > > >implementation details or API designs, though, so let's try to keep
> > > >the discussion at a somewhat high level, and leave the details for
> > > >specs and summit discussions. I do hope you will correct any
> > > >misunderstandings or misconceptions, because unwinding this as an
> > > >outside observer has been quite a challenge and it's likely I have
> > > >some details wrong.
> > > >
> > > >As I understand it, there are basically two ways to upload an image
> > > >to glance using the V2 API today. The "POST" API pushes the image's
> > > >bits through the Glance API server, and the "task" API instructs
> > > >Glance to download the image separately in the background. At one
> > > >point apparently there was a bug that caused the results of the two
> > > >different paths to be incompatible, but I believe that is now fixed.
> > > >However, the two separate APIs each have different issues that make
> > > >them unsuitable for DefCore.
> > > >
> > > >The DefCore process relies on several factors when designating APIs
> > > >for compliance. One factor is the technical direction, as communicated
> > > >by the contributor community -- that's where we tell them things
> > > >like "we plan to deprecate the Glance V1 API". In addition to the
> > > >technical direction, DefCore looks at the deployment history of an
> > > >API. They do not want to require deploying an API if it is not seen
> > > >as widely usable, and they look for some level of existing adoption
> > > >by cloud providers and distributors as an indication of that the
> > > >API is desired and can be successfully used. Because we have multiple
> > > >upload APIs, the message we're sending on technical direction is
> > > >weak right now, and so they have focused on deployment considerations
> > > >to resolve the question.
> > > 
> > > The task upload process you're referring to is the one that uses the
> > > `import` task, which allows you to download an image from an external
> > > source, asynchronously, and import it in Glance. This is the old
> > > `copy-from` behavior that was moved into a task.
> > > 
> > > The "fun" thing about this - and I'm sure other folks in the Glance
> > > community will disagree - is that I don't consider tasks to be a
> > > public API. That is to say, I would expect tasks to be an internal API
> > > used by cloud admins to perform some actions (bsaed on its current
> > > implementation). Eventually, some of these tasks could be triggered
> > > from the external API but as background operations that are triggered
> > > by the well-known public ones and not through the task API.
> > 
> > Does that mean it's more of an "admin" API?
> > 
> 
> I think it is basically just a half-way done implementation that is
> exposed directly to users of Rackspace Cloud and, AFAIK, nobody else.
> When last I tried to make integration tests in shade that exercised the
> upstream glance task import code, I was met with an implementation that
> simply did not work, because the pieces behind it had never been fully
> implemented upstream. That may have been resolved, but in the process
> of trying to write tests and make this work, I discovered a system that
> made very little sense from a user standpoint. I want to upload an
> image, why do I want a task?!
> 
> > > 
> > > Ultimately, I believe end-users of the cloud simply shouldn't care
> > > about what tasks are or aren't and more importantly, as you mentioned
> > > later in the email, tasks make clouds not interoperable. I'd be pissed
> > > if my public image service would ask me to learn about tasks to be
> > > able to use the service.
> > 
> > It would be OK if a public API set up to do a specific task returned a
> > task ID that could be used with a generic task API to check status, etc.
> > So the idea of tasks isn't completely bad, it's just too vague as it's
> > exposed right now.
> > 
> 
> I think it is a concern, because it is assuming users will want to do
> generic things with a specific API. This turns into a black-box game where
> the user shoves a task in and then waits to see what comes out the other
> side. Not something I want to encourage users to do or burden them with.
> 
> We have an API whose sole purpose is to accept image uploads. That
> Rackspace identified a scaling pain point there is _good_. But why not
> *solve* it for the user, instead of introduce more complexity?

That's fair. I don't actually care which API we have, as long as it
meets the other requirements.

> 
> What I'd like to see is the upload image API given the ability to
> respond with a URL that can be uploaded to using the object storage API
> we already have in OpenStack. Exposing users to all of these operator
> choices is just wasting their time. Just simply say "Oh, you want to
> upload an image? Thats fine, please upload it as an object over there
> and POST here again when it is ready to be imported." This will make
> perfect sense to a user reading docs, and doesn't require them to grasp
> an abstract concept like "tasks" when all they want to do is upload
> their image.
> 

And what would it do if the backing store for the image service
isn't Swift or another object storage system that supports direct
uploads? Return a URL that pointed back to itself, maybe?

Doug