Open Stack

> -----Original Message-----
> From: Doug Hellmann [mailto:doug at doughellmann.com]
> Sent: Monday, September 14, 2015 5:40 PM
> To: openstack-dev
> Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> 
> Excerpts from Kuvaja, Erno's message of 2015-09-14 15:02:59 +0000:
> > > -----Original Message-----
> > > From: Flavio Percoco [mailto:flavio at redhat.com]
> > > Sent: Monday, September 14, 2015 1:41 PM
> > > To: OpenStack Development Mailing List (not for usage questions)
> > > Subject: Re: [openstack-dev] [glance] proposed priorities for Mitaka
> > >
> > > On 14/09/15 08:10 -0400, Doug Hellmann wrote:
<CLIP>
> > > >
> > > >I. DefCore
> > > >
> > > >The primary issue that attracted my attention was the fact that
> > > >DefCore cannot currently include an image upload API in its
> > > >interoperability test suite, and therefore we do not have a way to
> > > >ensure interoperability between clouds for users or for trademark
> > > >use. The DefCore process has been long, and at times confusing,
> > > >even to those of us following it sort of closely. It's not entirely
> > > >surprising that some projects haven't been following the whole
> > > >time, or aren't aware of exactly what the whole thing means. I have
> > > >proposed a cross-project summit session for the Mitaka summit to
> > > >address this need for communication more broadly, but I'll try to
> summarize a bit here.
> > >
> >
> > Looking how different OpenStack based public clouds limits or fully
> prevents their users to upload images to their deployments, I'm not
> convinced the Image Upload should be included to this definition.
> 
> The problem with that approach is that it means end consumers of those
> clouds cannot write common tools that include image uploads, which is a
> frequently used/desired feature. What makes that feature so special that we
> don't care about it for interoperability?
> 

I'm not sure it really is so special API or technical wise, it's just the one that was lifted to the pedestal in this discussion.

> >
> > > +1
> > >
> > > I think it's quite sad that some projects, especially those
> > > considered to be part of the `starter-kit:compute`[0], don't follow
> > > closely what's going on in DefCore. I personally consider this a
> > > task PTLs should incorporate in their role duties. I'm glad you
> > > proposed such session, I hope it'll help raising awareness of this effort
> and it'll help moving things forward on that front.
> > >
> > >
> > > >
> > > >DefCore is using automated tests, combined with business policies,
> > > >to build a set of criteria for allowing trademark use. One of the
> > > >goals of that process is to ensure that all OpenStack deployments
> > > >are interoperable, so that users who write programs that talk to
> > > >one cloud can use the same program with another cloud easily. This
> > > >is a *REST
> > > >API* level of compatibility. We cannot insert cloud-specific
> > > >behavior into our client libraries, because not all cloud consumers
> > > >will use those libraries to talk to the services. Similarly, we
> > > >can't put the logic in the test suite, because that defeats the
> > > >entire purpose of making the APIs interoperable. For this level of
> > > >compatibility to work, we need well-defined APIs, with a long
> > > >support period, that work the same no matter how the cloud is
> > > >deployed. We need the entire community to support this effort. From
> > > >what I can tell, that is going to require some changes to the
> > > >current Glance API to meet the requirements. I'll list those
> > > >requirements, and I hope we can discuss them to a degree that
> > > >ensures everyone understands them. I don't want this email thread
> > > >to get bogged down in implementation details or API designs,
> > > >though, so let's try to keep the discussion at a somewhat high
> > > >level, and leave the details for specs and summit discussions. I do
> > > >hope you will correct any misunderstandings or misconceptions,
> > > >because unwinding this as an outside observer has been quite a
> challenge and it's likely I have some details wrong.
> >
> > This just reinforces my doubt above. By including upload to the defcore
> requirements probably just closes out lots of the public clouds out there. Is
> that the intention here?
> 
> No, absolutely not. The intention is to provide clear technical direction about
> what we think the API for uploading images should be.
> 

Gr8, that's easy goal to stand behind and support!

> >
> > > >
<CLIP>
> > >
> > > The task upload process you're referring to is the one that uses the
> > > `import` task, which allows you to download an image from an
> > > external source, asynchronously, and import it in Glance. This is
> > > the old `copy-from` behavior that was moved into a task.
> > >
> > > The "fun" thing about this - and I'm sure other folks in the Glance
> > > community will disagree - is that I don't consider tasks to be a
> > > public API. That is to say, I would expect tasks to be an internal
> > > API used by cloud admins to perform some actions (bsaed on its
> > > current implementation). Eventually, some of these tasks could be
> > > triggered from the external API but as background operations that
> > > are triggered by the well-known public ones and not through the task
> API.
> > >
> > > Ultimately, I believe end-users of the cloud simply shouldn't care
> > > about what tasks are or aren't and more importantly, as you
> > > mentioned later in the email, tasks make clouds not interoperable.
> > > I'd be pissed if my public image service would ask me to learn about tasks
> to be able to use the service.
> >
> > I'd like to bring another argument here. I think our Public Images API should
> behave consistently regardless if there is tasks enabled in the deployment or
> not and with what plugins. This meaning that _if_ we expect glance upload
> work over the POST API and that endpoint is available in the deployment I
> would expect a) my image hash to match with the one the cloud returns b)
> I'd assume all or none of the clouds rejecting my image if it gets flagged by
> Vendor X virus definitions and c) it being bootable across the clouds taken it's
> in supported format. On the other hand if I get told by the vendor that I need
> to use cloud specific task that accepts only ova compliant image packages and
> that the image will be checked before acceptance, my expectations are quite
> different and I would expect all that happening outside of the standard API
> as it's not consistent behavior.
> 
> I'm not sure what you're arguing. Is it not possible to have a background
> process import an image without modifying it?

Absolutely not my point, sorry. What I was trying to say here is that I rather have multiple ways of getting images into cloud I use if that means I can predict exactly how those different ways behaves on each deployment that has them enabled. My thought of tasks has been more on the processing side than just different way to do upload. So I have been in understanding that the intention behind tasks were to provide extra functionality like introspection, conversion, ova-import (which is under work currently), many of that would be beneficial for the end user of the cloud, but happening to your image as side effect of image upload without you knowing or able to avoid it, not so desirable state.

> 
> >
> > >
> > > Long story short, I believe the only upload API that should be
> > > considered is the one that uses HTTP and, eventually, to bring
> > > compatibility with v1 as far as the copy-from behavior goes, Glance
> > > could bring back that behavior on top of the task (just dropping
> > > this here for the sake of discussion and interoperability).
> > >
> > >
> > > >The POST API is enabled in many public clouds, but not consistently.
> > > >In some clouds like HP, a tenant requires special permission to use
> > > >the API. At least one provider, Rackspace, has disabled the API entirely.
> > > >This is apparently due to what seems like a fair argument that
> > > >uploading the bits directly to the API service presents a possible
> > > >denial of service vector. Without arguing the technical merits of
> > > >that decision, the fact remains that without a strong consensus
> > > >from deployers that the POST API should be publicly and
> > > >consistently available, it does not meet the requirements to be
> > > >used for DefCore testing.
> > >
> > > This is definitely unfortunate. I believe a good step forward for
> > > this discussion would be to create a list of issues related to
> > > uploading images and see how those issues can be addressed. The
> > > result from that work might be that it's not recommended to make
> > > that endpoint public but again, without going through the issues,
> > > it'll be hard to understand how we can improve this situation. I expect
> most of this issues to have a security impact.
> > >
> >
> > ++, regardless of the helpfulness of that discussion, I don't think it's realistic
> expectation to prioritize that work so much that majority of those issues
> would be solved amongst the priorities at the top of this e-mail within a cycle.
> 
> We should establish the priorities, even if the roadmap to solving them is
> longer than one cycle.
> 

That's fair enough.

> >
> > >
<CLICK>
> > >
> > > >Tasks also come from plugins, which may be installed differently
> > > >based on the deployment. This is an interesting approach to
> > > >creating API extensions, but isn't discoverable enough to write
> > > >interoperable tools against. Most of the other projects are
> > > >starting to move away from supporting API extensions at all because
> > > >of interoperability concerns they introduce. Deployers should be
> > > >able to configure their clouds to perform well, but not to behave in
> fundamentally different ways.
> > > >Extensions are just that, extensions. We can't rely on them for
> > > >interoperability testing.
> > >
> > > This is, indeed, an interesting interpretation of what tasks are for.
> > > I'd probably just blame us (Glance team) for not communicating
> > > properly what tasks are meant to be. I don't believe tasks are a way
> > > to extend the
> > > *public* API and I'd be curious to know if others see it that way. I
> > > fully agree that just breaks interoperability and as I've mentioned
> > > a couple of times in this reply already, I don't even think tasks should be
> part of the public API.
> >
> > Hmm-m, that's exactly how I have seen it. Plugins that can be provided to
> expand the standard functionality. I totally agree these not to being relied on
> interoperability. I've always assumed that that has been also the reason why
> tasks have not had too much focus as we've prioritized the actual API
> functionality and stability before it's expandability.
> 
> My issue with the task API as it stands is not how it works on the back-end.
> It's only the actual REST API entry point that concerns me, because of its
> vagueness.

That's great. This way the issues are easier to address than if the concerns were against the fundamental approach.

> 
> If there are different semantics for the background task in different
> providers because of different plugins, then we need to write tests to
> specify the aspects we care about being standardized. You mentioned image
> hashes as one case, and I can see that being useful. Rather than allowing the
> cloud to modify an image on upload, we might enforce that the hash of the
> image we download is the same as the hash of the image imported. That still
> allows for unpacking and scanning on the back-end.
> 

Again this is easy to back up.

> >
> > >
<CLIP>

- Erno