Open Stack

Keith, I was with you all the way till your last sentence ...

> -----Original Message-----
> From: Keith Bray [mailto:keith.bray at RACKSPACE.COM]
> Sent: Friday, April 22, 2016 11:53 AM
> To: OpenStack Development Mailing List (not for usage questions)
> <openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [magnum][app-catalog][all] Build unified
> abstraction for all COEs
> 
> Thanks Amirth… I’m glad to see it hasn’t changed much since I was involved
> with Trove in its early days.  What you are describing makes sense, and I
> view it as an LCD for managing common things across the database types,
> not an LCD for the database interaction performed by the user/client
> interacting with the application database.  This parallels where I think
> Magnum should sit, which is general management of the COEs (reinitialize
> bay, backup bay, maybe even common configuration of bays, etc. etc.), and
> not an LCD for user/application interaction with the COEs.  It’s a grey
> area for sure, as should “list containers” on a bay be a common
> abstraction?  I think it’s too early to tell… and, to be clear for all
> folks, I’m not opposed to the LCD existing.  I just don’t want it to be
> required for the operator to run it at this time as part of Magnum given
> how quickly the COE technology landscape is evolving.  So, optional
> support, or separate API/Project make the most sense to me, and can always
> be merged in as part of the Magnum project at a future date once the
> technology landscape settles.  RDBMS has been fairly standard for awhile.
> 

[amrith] Yes, but these pesky things called NoSQL have not, and Trove tries to handle both. Fun fact, there are more NoSQL databases than there are Ben & Jerry's flavors.

There is some of the same kind of challenges that you describe and while we currently have an API that tries to give you the LCD, I can certainly see the value in the approach you propose for Magnum. If there are going to be conversations about this in the magnum sessions, I would certainly like to attend as I think they will help to instruct us on how you are handling a similar class of challenges to ones that we face in Trove.

> Thanks for all the input.  The context helps.
> 
> -Keith
> 
> 
> 
> On 4/22/16, 6:40 AM, "Amrith Kumar" <amrith at tesora.com> wrote:
> 
> >For those interested in one aspect of this discussion (a common compute
> >API for bare-metal, VM's and containers), there's a review of a spec in
> >Trove [1], and a session at the summit [2].
> >
> >Please join [2] if you are able
> >
> >     Trove Container Support
> >     Thursday, April 28, 9:50am-10:30am
> >     Hilton Austin - MR 406
> >
> >Keith, more detailed answer to one of your questions is below.
> >
> >Thanks,
> >
> >-amrith
> >
> >
> >[1] https://review.openstack.org/#/c/307883/4
> >[2]
> >https://www.openstack.org/summit/austin-2016/summit-schedule/events/915
> >0
> >
> >> -----Original Message-----
> >> From: Keith Bray [mailto:keith.bray at RACKSPACE.COM]
> >> Sent: Thursday, April 21, 2016 5:11 PM
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> <openstack-dev at lists.openstack.org>
> >> Subject: Re: [openstack-dev] [magnum][app-catalog][all] Build unified
> >> abstraction for all COEs
> >>
> >> 100% agreed on all your points… with the addition that the level of
> >>functionality you are asking for doesn’t need to be baked into an API
> >>service such as Magnum.  I.e., Magnum doesn’t have to be the thing
> >>providing the easy-button app deployment — Magnum isn’t and shouldn’t
> >>be a  Docker Hub alternative, a Tutum alternative, etc.  A Horizon UI,
> >>App  Catalog UI, or OpenStack CLI on top of Heat, Murano, Solum,
> >>Magnum, etc.
> >> etc. can all provide this by pulling together the underlying API
> >> services/technologies to give users the easy app deployment buttons.
> I
> >> don’t think Magnum should do everything (or next thing we know we’ll
> >>be  trying to make Magnum a PaaS, or make it a CircleCI, or … Ok, I’ve
> >>gotten  carried away).  Hopefully my position is understood, and no
> >>problem if  folks disagree with me.  I’d just rather compartmentalize
> >>domain concerns  and scope Magnum to something focused, achievable,
> >>agnostic, and easy for  operators to adopt first. User traction will
> >>not be helped by increasing  service/operator complexity.  I’ll have
> >>to go look at the latest Trove and  Sahara APIs to see how LCD is
> >>incorporated, and would love feedback from
> >
> >[amrith] Trove provides a common, database agnostic set of API's for a
> >number of common database workflows including provisioning and
> >lifecycle management. It also provides abstractions for common database
> >topologies like replication and clustering, and management actions that
> >will manipulate those topologies (grow, shrink, failover, ...). It
> >provides abstractions for some common database administration
> >activities like user management, database management, and ACL's. It
> >allows you to take backups of databases and to launch new instances
> >from backups. It provides a simple way in which a user can manage the
> >configuration of databases (a subset of the configuration parameters
> >that the database supports, the choice the subset being up to the
> >operator) in a consistent way. Further it allows users to make
> >configuration changes across a group of databases through the process
> >of associating a 'configuration group' to database instances.
> >
> >The important thing about this is that there is a desire to provide all
> >of the above capabilities through the Trove API and make these
> >capabilities database agnostic. The actual database specific
> >implementations are within Trove and largely contained in a database
> >specific guest agent that performs the database specific actions to
> >achieve the end result that the user requested via the Trove API.
> >
> >The user interacts directly with the database as well; the application
> >speaks native database API's to the database and unlike (for example,
> >DynamoDB) Trove does not get into the data path between the application
> >and the database itself. Users and administrators are able to interact
> >with the database through its native management interfaces as well
> >(some restrictions may apply, depending on the level of access that the
> >operator allows).
> >
> >In short, the value provided is that databases are long lived things
> >and provisioning and initial configuration are very important, but
> >ongoing maintenance and management are as well. The mantra for dba's is
> >always to automate and standardize all the repeated workflows. Trove
> >does that for you through a single set of API's because todays
> >datacenters have a wide diversity of databases. Hope that helps.
> >
> >> Trove and Sahara operators on the value vs. customer confusion or
> >>operator  overhead they get from those LCDs if they are required parts
> >>of the  services.
> >>
> >> Thanks,
> >> -Keith
> >>
> >> On 4/21/16, 3:31 PM, "Fox, Kevin M" <Kevin.Fox at pnnl.gov> wrote:
> >>
> >> >There are a few reasons, but the primary one that affects me is Its
> >> >from the app-catalog use case.
> >> >
> >> >To gain user support for a product like OpenStack, you need users.
> >> >The easier you make it to use, the more users you can potentially get.
> >> >Traditional Operating Systems learned this a while back. Rather then
> >> >make each OS user have to be a developer and custom deploy every app
> >> >they want to run, they split the effort in such a way that
> >> >Developers can provide software through channels that Users that are
> >> >not skilled Developers can consume and deploy. The "App" culture in
> >> >the mobile space it the epitome of that at the moment. My
> >> >grandmother fires up the app store on her phone, clicks install on
> >> >something interesting, and
> >> starts using it.
> >> >
> >> >Right now, Thats incredibly difficult in OpenStack. You have to find
> >> >the software your interested in, figure out which components your
> >> >going to consume (nova, magnum, which COE, etc) then use those api's
> >> >to launch some resource. Then after that resource is up, then you
> >> >have to switch tools and then use those tools to further launch
> >> >things, ansible or kubectl or whatever, then further deploy things.
> >> >
> >> >What I'm looking for, is a unified enough api, that a user can go
> >> >into horizon, go to the app catalog, find an interesting app, click
> >> >install/run, and then get a link to a service they can click on and
> >> >start consuming the app they want in the first place. The number of
> >> >users that could use such an interface, and consume OpenStack
> >> >resources are several orders of magnitude greater then the numbers
> >> >that can manually deploy something ala the procedure in the previous
> paragraph.
> >> >More of that is good for Users, Developers, and Operators.
> >> >
> >> >Does that help?
> >> >
> >> >Thanks,
> >> >Kevin
> >> >
> >> >
> >> >________________________________________
> >> >From: Keith Bray [keith.bray at RACKSPACE.COM]
> >> >Sent: Thursday, April 21, 2016 1:10 PM
> >> >To: OpenStack Development Mailing List (not for usage questions)
> >> >Subject: Re: [openstack-dev] [magnum][app-catalog][all] Build
> >> >unified abstraction for all COEs
> >> >
> >> >If you don¹t want a user to have to choose a COE, can¹t we just
> >> >offer an option for the operator to mark a particular COE as the
> >> >³Default COE² that could be defaulted to if one isn¹t specified in
> >> >the Bay create call?  If the operator didn¹t specify a default one,
> >> >then the CLI/UI must submit one in the bay create call otherwise it
> would fail.
> >> >
> >> >Kevin, can you clarify Why you have to write scripts to deploy a
> >> container
> >> >to the COE?   It can be made easy for the user to extract all the
> >> >runtime/env vars needed for a user to just do ³docker run Š²  and
> >> >poof, container running on Swarm on a Magnum bay.  Can you help me
> understand
> >> >the script part of it?   I don¹t believe container users want an
> >> >abstraction between them and their COE CLIŠ but, what I believe
> >> >isn¹t important.  What I do think is important is that we not
> >> >require OpenStack operators to run that abstraction layer to be
> >> >running a ³magnum compliant² service.  It should either be an
> >> >³optional² API add-on or a separate API or separate project.  If
> >> >some folks want an abstraction layer, then great, feel free to build
> >> >it and even propose
> >>it
> >> under the OpenStack ecosystem..
> >> >But, that abstraction would be a ³proxy API" over the COEs, and
> >> >doesn¹t need to be part of Magnum¹s offering, as it would be
> >> >targeted at the COE interactions and not the bay interactions (which
> >> >is where Magnum scope is best focused).  I don¹t think Magnum should
> >> >play in both these distinct domains (Bay interaction vs. COE
> >> >interaction).  The former (bay
> >> >interaction) is an infrastructure cloud thing (fits well with
> >> >OpenStack), the latter (COE interaction) is an obfuscation of
> >> >emerging technologies, which gets in to the Trap that Adrian
> >> >mentioned.  The abstraction layer API will forever and always be
> >> >drastically behind in trying to keep up with the COE innovation.
> >> >
> >> >In summary, an abstraction over the COEs would be best served as a
> >> >different effort.  Magnum would be best focused on bay interactions
> >> >and should not try to pick a COE winner or require an operator to
> >> >run a lowest-common-demonitor API abstraction.
> >> >
> >> >Thanks for listening to my soap-box.
> >> >-Keith
> >> >
> >> >
> >> >
> >> >On 4/21/16, 2:36 PM, "Fox, Kevin M" <Kevin.Fox at pnnl.gov> wrote:
> >> >
> >> >>I agree with that, and thats why providing some bare minimum
> >> >>abstraction will help the users not have to choose a COE themselves.
> >> >>If we can't decide, why can they? If all they want to do is launch
> >> >>a container, they should be able to script up "magnum
> >> >>launch-container foo/bar:latest" and get one. That script can then be
> relied upon.
> >> >>
> >> >>Today, they have to write scripts to deploy to the specific COE
> >> >>they have chosen. If they chose Docker, and something better comes
> >> >>out, they have to go rewrite a bunch of stuff to target the new,
> >> >>better thing. This puts a lot of work on others.
> >> >>
> >> >>Do I think we can provide an abstraction that prevents them from
> >> >>ever having to rewrite scripts? No. There are a lot of features in
> >> >>the COE world in flight right now and we dont want to solidify an
> >> >>api around them yet. We shouldn't even try that. But can we cover a
> >> >>few common things now? Yeah.
> >> >>
> >> >>Thanks,
> >> >>Kevin
> >> >>________________________________________
> >> >>From: Adrian Otto [adrian.otto at rackspace.com]
> >> >>Sent: Thursday, April 21, 2016 7:32 AM
> >> >>To: OpenStack Development Mailing List (not for usage questions)
> >> >>Subject: Re: [openstack-dev] [magnum][app-catalog][all] Build
> >> >>unified abstraction for all COEs
> >> >>
> >> >>> On Apr 20, 2016, at 2:49 PM, Joshua Harlow
> >> >>><harlowja at fastmail.com>
> >> >>>wrote:
> >> >>>
> >> >>> Thierry Carrez wrote:
> >> >>>> Adrian Otto wrote:
> >> >>>>> This pursuit is a trap. Magnum should focus on making native
> >> >>>>>container  APIs available. We should not wrap APIs with leaky
> >> >>>>>abstractions. The  lowest common denominator of all COEs is an
> >> >>>>>remarkably low value API  that adds considerable complexity to
> >> >>>>>Magnum that will not  strategically advance OpenStack. If we
> >> >>>>>instead focus our effort on  making the COEs work better on
> >> >>>>>OpenStack, that would be a winning  strategy. Support and
> >> >>>>>compliment our various COE ecosystems.
> >> >>>
> >> >>> So I'm all for avoiding 'wrap APIs with leaky abstractions' and
> >> >>>'making  COEs work better on OpenStack' but I do dislike the part
> >> >>>about COEs
> >> >>>(plural) because it is once again the old non-opinionated problem
> >> >>>that we (as a community) suffer from.
> >> >>>
> >> >>> Just my 2 cents, but I'd almost rather we pick one COE and
> >> >>>integrate that deeply/tightly with openstack, and yes if this
> >> >>>causes some part of the openstack community to be annoyed, meh, to
> >> >>>bad. Sadly I have a feeling we are hurting ourselves by continuing
> >> >>>to try to be everything and not picking anything (it's a general
> >> >>>thing we, as a group, seem to be good at, lol). I mean I get the
> >> >>>reason to just support all the things, but it feels like we as a
> >> >>>community could just pick something, work together on figuring out
> >> >>>how to pick one, using all these bright leaders we have to help
> >> >>>make that possible (and yes this might piss some people off, to
> >> >>>bad). Then work toward making that something great and move onŠ
> >> >>
> >> >>The key issue preventing the selection of only one COE is that this
> >> >>area is moving very quickly. If we would have decided what to pick
> >> >>at the time the Magnum idea was created, we would have selected
> Docker.
> >> >>If you look at it today, you might pick something else. A few
> >> >>months down the road, there may be yet another choice that is more
> >> >>compelling. The fact that a cloud operator can integrate services
> >> >>with OpenStack, and have the freedom to offer support for a
> >> >>selection of COE¹s is a form of insurance against the risk of
> >> >>picking the wrong one. Our compute service offers a choice of
> >> >>hypervisors, our block storage service offers a choice of storage
> >> >>hardware drivers, our networking service allows a choice of network
> >> >>drivers. Magnum is following the same pattern of choice that has
> >> >>made OpenStack compelling for a very diverse community. That design
> >> >>consideration was
> >> intentional.
> >> >>
> >> >>Over time, we can focus the majority of our effort on deep
> >> >>integration with COEs that users select the most. I¹m convinced
> >> >>it¹s still too early to bet the farm on just one choice.
> >> >>
> >> >>Adrian
> >> >>
> >> >>>> I'm with Adrian on that one. I've attended a lot of
> >> >>>>container-oriented  conferences over the past year and my main
> >> >>>>takeaway is that this new  crowd of potential users is not
> >> >>>>interested (at all) in an  OpenStack-specific lowest common
> >> >>>>denominator API for COEs. They want to  take advantage of the
> >> >>>>cool features in Kubernetes API or the versatility  of Mesos.
> >> >>>>They want to avoid caring about the infrastructure provider  bit
> >> >>>>(and not deploy Mesos or Kubernetes themselves).
> >> >>>>
> >> >>>> Let's focus on the infrastructure provider bit -- that is what
> >> >>>>we do and  what the ecosystem wants us to provide.
> >> >>>>
> >> >>>
> >> >>>
> >> >>>__________________________________________________________________
> >> >>>___
> >> >>>___
> >> >>>_
> >> >>>_
> >> >>> OpenStack Development Mailing List (not for usage questions)
> >> >>> Unsubscribe:
> >> >>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> >>___________________________________________________________________
> >> >>___
> >> >>___
> >> >>_
> >> >>OpenStack Development Mailing List (not for usage questions)
> >> >>Unsubscribe:
> >> >>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >>
> >> >>___________________________________________________________________
> >> >>___
> >> >>___
> >> >>_
> >> >>OpenStack Development Mailing List (not for usage questions)
> >> >>Unsubscribe:
> >> >>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> >
> >> >____________________________________________________________________
> >> >___ ___ OpenStack Development Mailing List (not for usage questions)
> >> >Unsubscribe:
> >> >OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >> >
> >> >____________________________________________________________________
> >> >___ ___ OpenStack Development Mailing List (not for usage questions)
> >> >Unsubscribe:
> >> >OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>______________________________________________________________________
> >>___
> >>_
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >_______________________________________________________________________
> >___ OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe:
> >OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev