Open Stack

On 10 February 2016 at 21:37, Mark Voelker <mvoelker at vmware.com> wrote:
> Hi John,
>
> Thanks for writing this up John!  A few comments:
>
>> On Feb 10, 2016, at 1:25 PM, John Garbutt <john at johngarbutt.com> wrote:
>>
>> Hi,
>>
>> DefCore, in its current form, is doing a great job of defining things
>> that should work on all OpenStack deployments. Stopping the divergence
>> of OpenStack APIs is super important, and this is working.
>>
>> But I think we are hitting issues with getting projects engaged on
>> driving forward the cause of interoperability, and its proving tricky
>> to expand beyond the base set of required operations.
>
> I’ll just chime in here to say that with my DefCore hat on, I’m interested to hear feedback from devs on those projects as to why this is.  =)

I think we need to talk about problems that need to be solved. That
will engage people. I kinda talk about that a bit in here:
https://www.openstack.org/summit/tokyo-2015/videos/presentation/working-with-upstream-openstack-deadlines-and-internal-deadlines

It can feel like "legal stuff that doesn't concern me".

In reality, time has been my main blocker. Spotting the networking and
volume things getting adding really confuses me, hence me getting
engaged again with the discussion.

But thats maybe a separate thread...

>> If we look a few years ahead, I would love to see a broad ecosystem of
>> applications written to run against any certified OpenStack Compute
>> cloud. In addition, some applications may need some additional
>> capabilities (configure network routers, needs object storage, etc). I
>> am thinking about the next generation of this:
>> http://apps.openstack.org/
>>
>> I think DefCore can help make that dream become a reality.
>>
>> To reach that goal more quickly, I think should change direction
>> slight, and consider these three ideas:
>>
>> 1: Focus on Uses cases
>>
>> Lets define the use cases application developers need for the above
>> vision to become reality. Thats helps projects engage with the
>> problems that need solving. Sometimes the API exists, sometimes there
>> is a possible workflow to do this in an interoperable way, other times
>> a new API or APIs might be needed. These patterns can then be used to
>> create applications that work across all OpenStack certified clouds. I
>> think the product working group may be able to help.
>>
>
> +1 for considering use cases.  This is something I’ve been kicking around a bit myself but haven’t managed to write up yet.  I have admittedly not been thinking about looking at use cases through the lens of the Community App Catalog though, for reasons I’ll get to a bit further down. =)
>
> Instead, I’ve been thinking about the use cases for which OpenStack seems to be frequently deployed for today per the user survey (and other industry reports).  For those reading along who aren’t familiar, the user survey has some questions about what workloads, frameworks, and stacks are running on top of OpenStack.  As an example, the most popular stack in the last survey was the LAMP stack by a large margin, and the most popular workload was “Software dev/test/QA and CI” by a substantial margin.  However the findings are sort of…generic?...and therefore a bit tricky to use as actionable intelligence, but are a good starting point nonetheless in that they give us some areas of focus if not specifics.  Certainly there are some well known users of OpenStack in the community who can help fill in details of how they’re using OpenStack for those things (our own infra team being one example, and one that we’ve looked at in the past…I know I for one have perused shade’s code more than once, for instance).
>
> That said, I also want to be careful to point out that I’d like for us not to simply focus on use cases people are deploying OpenStack for today, but also what opportunities we’d uncork tomorrow if we solved a few problems.  Gauging that involves a little bit of gazing into the old crystal ball, but I think it’s a worthwhile exercise and I’d be happy to hear inputs.  Since I’ve already mentioned the user survey, I should also point out that it already has some questions related to what new/emerging tech users are interested in (containers, NFV, PaaS, IoT, etc).  Again: sort of generic, but perhaps a starting point.
>

+1 all that.

Just running an HA LAMP stack and Zuul (or just Jenkins) would get us
a long way forward.


>> 2: Validating Optional Capabilities
>>
>> While its important to define "things that should work everywhere" I
>> think its important to define "if available, it should work the same
>> way everywhere". This requires that the project APIs work on policy
>> discovery, so you can tell if an API is available in a particular
>> deployment or not. Its possible we end up defining the minimum
>> standard as having one of n choices available (assuming there is a
>> workflow that lets you pick between them, ideally an abstract API
>> would hide that complexity, but thats not always possible).
>
>
> There’s been some discussion around this in the past in a couple of different lights.  First, API and policy discoverability has come up as a pain point more than once.  At the next DefCore midcycle we’ll be starting to plot out the first report on major barriers to interoperability that we intend to deliver periodically to the TC/UC, and this is one that’s squarely on my candidate list for inclusion.


Its well understood, and a long standing TODO, at least in Nova. Its
certainly on our mind for Newton, as there are things we don't want to
add without policy discovery.
The problem is agreeing how to fix.


> Second, we’ve also had interest from some in the community in having a sort of comparison matrix of supported features.  The OpenStack Powered (TM) logo gives you some level of interoperability at a glance, which is simple and good.  But you might also be interested in features that are less frequently included and want to know what clouds/products support them.  One concrete example: we had a request a while back to show what products supported EC2 compatibility (and to what degree, by showing a set of test results in RefStack).  We’ve also had suggestions that there might need to be a special set of capabilities or even a special logo program for specialized use cases (the most oft-sited one being NFV).  That sort of thing kind of gets to the general notion of “if available, it should work the same way AND I have a way to figure out what products support the things I actually need whether regardless of whether they’re ‘core' or not".
>
> To some degree I think those sorts of things haven’t yet come to fruition in part because there’s been a focus on getting a solid set of capabilities to define “core OpenStack” before moving on to adjacent things…but we’re now a good deal further along with that than we were even six months ago (we have advisory networking capabilities for the first time!  And etc.), so perhaps it’s time to revisit.  Naturally these things don’t come without costs so we’d need to line up some manandwomanpower to help…
>
> */me smiles politely and winks at everyone reading this email*


So I think policy discovery is the dependency here.

FWIW, we are trying to be more precise about what features work where in Nova:
http://docs.openstack.org/developer/nova/feature_classification.html

I hope the groupings used above could be adopted by DefCore for
optional features.


>> 3. Working with App Developers to test this
>>
>> One idea to make this all real is to look at the existing apps in the
>> marketplace app store,
>
> Hmmm.

Actually, yeah not many good candidates.

An HA LAMP stack would do just fine.
I just want to start with *real* things.

> All that said: I’m not trying to pick on the Community App Catalog here at all.  It’s still in beta, so I’m merely wondering if what’s there today is really a useful indicative set, or if instead it’s something we should revisit when it’s a bit more mature.  I think your bigger point, though, is how to get to a place where people want to create the templates/packages/images they contribute to the Community App Catalog to be able to run on more OpenStack clouds and have good guidance on how to do so.  That’s a fantastic goal, I’m just not sure if what’s in the app catalog today is the right starting point.
>


I think we are in agreement. The aim is the key bit.
The current approach has good goals and aspirations.
Its just not feasible with our current APIs.


>> and look at ways to make them work with all
>> certified OpenStack deployments/products. I think this is likely to
>> involve working out how to bootstrap their images from an existing
>> base image in that deployment, and then maybe then share that image
>> with other users of that clouds. (FWIW, I feel importing a full image
>> avoids any work each cloud has done to optimise their particular
>> environment, and many other issues).  It might be certain apps are
>> supported on particular certified products, but thats not the aim.  The
>> aim is to get apps that users can run on all certified products.
>
> I’m curious what the mechanics of this might look like.  For example: are you thinking of a sort of CI approach, where when a new app is published to the community app catalog each vendor sets up a CI system that tries to deploy it to their OpenStack Powered (TM) product and reports whether it works or not?  Or coming at it from the other direction, perhaps we work out a way to figure out which API’s and other capabilities the app relies on and then compare that to a listing of what each product claims to support?  Or perhaps something else?
>

I was thinking a DevStack setup that is setup such that only DefCore
tests APIs work, all others error out somehow.

Vendor has to validate their product works with that setup.


>> What do you all think? Does this look like the seeds of a good idea?
>> Or is there something I am totally missing here?
>
>
> I think you’ve raised several good points for discussion—thanks!

Happy to help kick off the discussion.

Thanks,
John