Open Stack

Hi John,

Thanks for writing this up John!  A few comments:

> On Feb 10, 2016, at 1:25 PM, John Garbutt <john at johngarbutt.com> wrote:
> 
> Hi,
> 
> DefCore, in its current form, is doing a great job of defining things
> that should work on all OpenStack deployments. Stopping the divergence
> of OpenStack APIs is super important, and this is working.
> 
> But I think we are hitting issues with getting projects engaged on
> driving forward the cause of interoperability, and its proving tricky
> to expand beyond the base set of required operations.

I’ll just chime in here to say that with my DefCore hat on, I’m interested to hear feedback from devs on those projects as to why this is.  =)  

> 
> If we look a few years ahead, I would love to see a broad ecosystem of
> applications written to run against any certified OpenStack Compute
> cloud. In addition, some applications may need some additional
> capabilities (configure network routers, needs object storage, etc). I
> am thinking about the next generation of this:
> http://apps.openstack.org/
> 
> I think DefCore can help make that dream become a reality.
> 
> To reach that goal more quickly, I think should change direction
> slight, and consider these three ideas:
> 
> 1: Focus on Uses cases
> 
> Lets define the use cases application developers need for the above
> vision to become reality. Thats helps projects engage with the
> problems that need solving. Sometimes the API exists, sometimes there
> is a possible workflow to do this in an interoperable way, other times
> a new API or APIs might be needed. These patterns can then be used to
> create applications that work across all OpenStack certified clouds. I
> think the product working group may be able to help.
> 

+1 for considering use cases.  This is something I’ve been kicking around a bit myself but haven’t managed to write up yet.  I have admittedly not been thinking about looking at use cases through the lens of the Community App Catalog though, for reasons I’ll get to a bit further down. =)

Instead, I’ve been thinking about the use cases for which OpenStack seems to be frequently deployed for today per the user survey (and other industry reports).  For those reading along who aren’t familiar, the user survey has some questions about what workloads, frameworks, and stacks are running on top of OpenStack.  As an example, the most popular stack in the last survey was the LAMP stack by a large margin, and the most popular workload was “Software dev/test/QA and CI” by a substantial margin.  However the findings are sort of…generic?...and therefore a bit tricky to use as actionable intelligence, but are a good starting point nonetheless in that they give us some areas of focus if not specifics.  Certainly there are some well known users of OpenStack in the community who can help fill in details of how they’re using OpenStack for those things (our own infra team being one example, and one that we’ve looked at in the past…I know I for one have perused shade’s code more than once, for instance).

That said, I also want to be careful to point out that I’d like for us not to simply focus on use cases people are deploying OpenStack for today, but also what opportunities we’d uncork tomorrow if we solved a few problems.  Gauging that involves a little bit of gazing into the old crystal ball, but I think it’s a worthwhile exercise and I’d be happy to hear inputs.  Since I’ve already mentioned the user survey, I should also point out that it already has some questions related to what new/emerging tech users are interested in (containers, NFV, PaaS, IoT, etc).  Again: sort of generic, but perhaps a starting point.


> 2: Validating Optional Capabilities
> 
> While its important to define "things that should work everywhere" I
> think its important to define "if available, it should work the same
> way everywhere". This requires that the project APIs work on policy
> discovery, so you can tell if an API is available in a particular
> deployment or not. Its possible we end up defining the minimum
> standard as having one of n choices available (assuming there is a
> workflow that lets you pick between them, ideally an abstract API
> would hide that complexity, but thats not always possible).


There’s been some discussion around this in the past in a couple of different lights.  First, API and policy discoverability has come up as a pain point more than once.  At the next DefCore midcycle we’ll be starting to plot out the first report on major barriers to interoperability that we intend to deliver periodically to the TC/UC, and this is one that’s squarely on my candidate list for inclusion.  

Second, we’ve also had interest from some in the community in having a sort of comparison matrix of supported features.  The OpenStack Powered (TM) logo gives you some level of interoperability at a glance, which is simple and good.  But you might also be interested in features that are less frequently included and want to know what clouds/products support them.  One concrete example: we had a request a while back to show what products supported EC2 compatibility (and to what degree, by showing a set of test results in RefStack).  We’ve also had suggestions that there might need to be a special set of capabilities or even a special logo program for specialized use cases (the most oft-sited one being NFV).  That sort of thing kind of gets to the general notion of “if available, it should work the same way AND I have a way to figure out what products support the things I actually need whether regardless of whether they’re ‘core' or not".

To some degree I think those sorts of things haven’t yet come to fruition in part because there’s been a focus on getting a solid set of capabilities to define “core OpenStack” before moving on to adjacent things…but we’re now a good deal further along with that than we were even six months ago (we have advisory networking capabilities for the first time!  And etc.), so perhaps it’s time to revisit.  Naturally these things don’t come without costs so we’d need to line up some manandwomanpower to help…

*/me smiles politely and winks at everyone reading this email* 


> 
> 3. Working with App Developers to test this
> 
> One idea to make this all real is to look at the existing apps in the
> marketplace app store,

Hmmm.  

By my count there’s about 40 Murano packages, 6 Heat templates, and 39 Glance images in the Community App Catalog today.  Some of those aren’t really apps in a business-problem-solving sense; they’re things like generic Debian or CentOS images, a Hello World Heat template, or things like MySQL packages that generally would be one part of an application (say, an e-commerce app) rather than a full app of it’s own right.  All of the Murano packages are built for Kilo or newer...recall that in the last user survey Juno and Icehouse were still with a few points of Kilo in terms of popularity and were more popular in production environments, so there’s some lag in getting OpenStack releases to production to consider.  Murano itself is used in only about 4% of production installs per the last user survey.  I also don’t think there’s a good way to see which apps in the catalog are popular and which are rarely ever downloaded.  Mapping apps in the Community App Catalog to use cases is also a bit tricky in that many of the packages there are sort of components of a full app or use case…for instance, I can get a Jenkins package from the catalog and I can get a LAMP stack glance image and I can get Chef server Heat template, but I have to cobble those and a few more things together to actually build a CI system to deploy and test the e-commerce app I’m developing (which is all maps back to the aforementioned "Software dev/test/QA and CI” workload from the user survey).  Anecdotally, a lot of the customers I’ve worked with over the past few years are building in-house apps on top of their clouds (I’ll mention e-commerce here again…you probably don’t see a lot of the of application code behind your favorite e-tailer’s website in the Community App Catalog, though you might see some of it’s underpinning components like databases, OS images, or CI tools).

All that said: I’m not trying to pick on the Community App Catalog here at all.  It’s still in beta, so I’m merely wondering if what’s there today is really a useful indicative set, or if instead it’s something we should revisit when it’s a bit more mature.  I think your bigger point, though, is how to get to a place where people want to create the templates/packages/images they contribute to the Community App Catalog to be able to run on more OpenStack clouds and have good guidance on how to do so.  That’s a fantastic goal, I’m just not sure if what’s in the app catalog today is the right starting point.

> and look at ways to make them work with all
> certified OpenStack deployments/products. I think this is likely to
> involve working out how to bootstrap their images from an existing
> base image in that deployment, and then maybe then share that image
> with other users of that clouds. (FWIW, I feel importing a full image
> avoids any work each cloud has done to optimise their particular
> environment, and many other issues).  It might be certain apps are
> supported on particular certified products, but thats not the aim.  The
> aim is to get apps that users can run on all certified products.

I’m curious what the mechanics of this might look like.  For example: are you thinking of a sort of CI approach, where when a new app is published to the community app catalog each vendor sets up a CI system that tries to deploy it to their OpenStack Powered (TM) product and reports whether it works or not?  Or coming at it from the other direction, perhaps we work out a way to figure out which API’s and other capabilities the app relies on and then compare that to a listing of what each product claims to support?  Or perhaps something else?


> 
> What do you all think? Does this look like the seeds of a good idea?
> Or is there something I am totally missing here?


I think you’ve raised several good points for discussion—thanks!

At Your Service,

Mark T. Voelker

> 
> Thanks,
> johnthetubaguy
> 
> _______________________________________________
> Defcore-committee mailing list
> Defcore-committee at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/defcore-committee