Open Stack

On Sun, 2016-01-24 at 15:42 -0800, Clint Byrum wrote:
> Excerpts from Dan Prince's message of 2016-01-22 16:19:07 -0800:
> > On Fri, 2016-01-22 at 11:24 -0600, Ben Nemec wrote:
> > > So I haven't weighed in on this yet, in part because I was on
> > > vacation
> > > when it was first proposed and missed a lot of the initial
> > > discussion,
> > > and also because I wanted to take some time to order my thoughts
> > > on
> > > it.
> > >  Also because my initial reaction...was not conducive to calm and
> > > rational discussion. ;-)
> > > 
> > > The tldr is that I don't like it.  To explain why, I'm going to
> > > make
> > > a
> > > list (everyone loves lists, right? Top $NUMBER reasons we should
> > > stop
> > > expecting other people to write our API for us):
> > > 
> > > 1) We've been down this road before.  Except last time it was
> > > with
> > > Heat.
> > >  I'm being somewhat tongue-in-cheek here, but expecting a general
> > > service to provide us a user-friendly API for our specific use
> > > case
> > > just
> > > doesn't make sense to me.
> > 
> > We've been down this road with Heat yes. But we are currently using
> > Heat for some things that we arguable should be (a workflows tool
> > might
> > help offload some stuff out of Heat). Also we haven't implemented
> > custom Heat resources for TripleO either. There are mixed opinions
> > on
> > this but plugging in your code to a generic API is quite nice
> > sometimes.
> > 
> > That is the beauty of Mistral I think. Unlike Heat it actually
> > encourages you to customize it with custom Python actions. Anything
> > we
> > want in tripleo-common can become our own Mistral action (these get
> > registered with stevedore entry points so we'd own the code) and
> > the
> > YAML workflows just tie them together via tasks.
> > 
> 
> Since TripleO's undercloud is a special, single-purpose cloud just
> for
> deploying OpenStack, I think that is a perfectly fine idea. But it
> also
> somewhat invalidates the reasons to use Mistral. If you can control
> all
> the pieces, IMO you may want to use the best tool for that job, not
> the
> one that is the most OpenStack. The virtuous circle only works if
> you're
> actually improving the OpenStack components for their general use
> cases.

Very true. And I think the goal would be that in general we try to
improve OpenStack components first. We don't want custom actions for
everything. Ideally core OpenStack services would support what we need
instead of us having to roll our own actions for things. However that
is sometimes easier said than done so we need to have a "release valve"
to move forwards.

Again, this is why I really like Mistral. It integrates directly with a
slew of OpenStack services by simply providing a lightweight glue
wrapper on top of the various python-* clients:

http://git.openstack.org/cgit/openstack/mistral/tree/mistral/actions/op
enstack/mapping.json

So yeah, instead of creating our own massive pile of custom TripleO
actions lets first ask the question does this code actually belong in
the upstream python client library? And would putting it there be a
generally useful addition to everyone?

Using a combination of both these approaches however I think we could
quickly move forwards and build the deployment workflows we need in the
realm of generic workflow tools.

Dan


> 
> > We don't have to go off and build our own proxy deployment workflow
> > API. The structure to do just about anything we need already exists
> > so
> > why not go and use it?
> > 
> > > 
> > > 2) The TripleO API is not a workflow API.  I also largely missed
> > > this
> > > discussion, but the TripleO API is a _Deployment_ API.  In some
> > > cases
> > > there also happens to be a workflow going on behind the scenes,
> > > but
> > > honestly that's not something I want our users to have to care
> > > about.
> > 
> > Agree that users don't have to care about this.
> > 
> > Users can get as involved as they want here. Most users I think
> > will
> > use python-tripleoclient to drive the deployment or the new UI.
> > They
> > don't have to interact with Mistral directly unless they really
> > want
> > to. So whether we choose to build our own API or use a generic one
> > I
> > think this point is mute.
> > 
> > > 
> > > 3) It ties us 100% to a given implementation.  If Mistral proves
> > > to
> > > be a
> > > poor choice for some reason, or insufficient for a particular use
> > > case,
> > > we have no alternative.  If we have an API and decide to change
> > > our
> > > implementation, nobody has to know or care.  This is kind of the
> > > whole
> > > point of having an API - it shields users from all the nasty
> > > implementation details under the surface.
> > 
> > 
> > Mistal's API is a generic workflow API. It is very much the same
> > layer
> > that I think we would get if we were to integrate with something
> > like
> > Ansible Tower... except that Mistral is part of OpenStack. It
> > integrates very nicely with OpenStack services and is very
> > customizable
> > with custom actions. The fact that Mistral sits much closer to
> > OpenStack and is essentially a light shim on top of it is to our
> > advantage (being TripleO). To think that we can build up a proxy
> > API in
> > such a manner that we might be able to swap in an entirely new
> > backend
> > (without even having a fully implement backend yet to begin with)
> > is
> > for me a bit of a stretch. We've got a lot of "TripleO API"
> > maturing
> > before we'll get to this point. Which is why I lean towards using a
> > generic workflow API to accomplis the same task.
> > 
> 
> Ansible 2.0 integrates extremely well with OpenStack too. Even
> better,
> you don't need an inside-the-cloud service to use it to interact
> with OpenStack. Now, sometimes you want an "aaS" helping you, and as
> you've alluded to, Tower is sort of like that. As is Mistral for its
> language. I've suggested before that Mistral would be well served by
> adopting Ansible as a language for workflow, since the concepts
> they're
> encoding are very similar. In fact, learning Mistral's DSL feels like
> learning Spanish after already having studied Latin. The structure is
> the same, the meanings are the same. It's just different words.
> 
> > I actually think rather than shielding users we should be more
> > transparent about the actual workflows that are driving deployment.
> > Smaller more focused workflows that we string together to drive the
> > deployment.
> > 
> 
> I tend to agree, especially when your workflows may need to be
> customized.
> 
> > > 
> > > 4) It raises the bar even further for both new deployers and
> > > developers.
> > >  You already need to have a pretty firm grasp of Puppet and Heat
> > > templates to understand how our stuff works, not to mention a
> > > decent
> > > understanding of quite a number of OpenStack services.
> > > 
> > > This presents a big chicken and egg problem for people new to
> > > OpenStack.
> > >  It's great that we're based on OpenStack and that allows people
> > > to
> > > peek
> > > under the hood and do some tinkering, but it can't be required
> > > for
> > > everyone.  A lot of our deployers are going to have little to no
> > > OpenStack experience, and TripleO is already a daunting task for
> > > those
> > > people (hell, it's daunting for people who _are_ experienced).
> > > 
> > 
> > And on the flipside you will get more of a community around using
> > an
> > OpenStack project than you ever would going off and building your
> > own
> > "Deployment/Workflow API". 
> > 
> 
> I actually think you'll get an even bigger community if you use a
> generic deployment workflow tool than if you use an openstack
> specific
> workflow tool. So IMO, the end-goal should be an API which drives
> Ansible workflows. Mistral would likely be able to do this with some
> refactoring around multiple languages.
> 
> > I would actually argue this is less of a deployers thing and more
> > of a
> > development tool choice. IMO most deployers will use python-
> > tripleoclient or some UI and not mistralclient directly. The code
> > I've
> > posted this week shows a prototype of just this, Mistral is swapped
> > in
> > such that you would never know it was involved because python-
> > tripleoclient works like it always did. Deployers use our CLI and
> > UI
> > tools like they always have, and developers gain a community of
> > Mistral
> > developers (and documentation) which they can interact with on
> > common
> > problems. Sounds like a win/win to me.
> > 
> > > 5) What does reimplementing all of our tested, well-understood
> > > Python
> > > into a new YAML format gain us?  This is maybe the biggest thing
> > > I'm
> > > missing from this whole discussion.  We lose a bunch of things
> > > (ease
> > > of
> > > transition from other Python projects, excellent existing testing
> > > framework, etc.), but what are we actually gaining other than the
> > > ability to say that we use N + 1 OpenStack services?  Because
> > > we're
> > > way
> > > past the point where "It's OpenStack deploying OpenStack" is
> > > sufficient
> > > reason for people to pay attention to us.  We need less "Ooh,
> > > neat"
> > > and
> > > more "Ooh, that's easy to use and works well."  It's still not
> > > clear
> > > to
> > > me that Mistral helps in any way with the latter.
> > 
> > Nobody suggested we reimplement everything. Much of the plan to
> > move
> > code into tripleo-common would stay. Instead of building our own
> > API
> > we'd just skip all that and focus on the code that is actually
> > about
> > our deployments in the form of custom Mistral actions and YAML
> > workflows.
> > 
> > The YAML workflows just ties together actions which are actually
> > all
> > written in Python. YAML works quite well for this and is a whole
> > lot
> > less verbose than writting everything we have in Python. There is a
> > reason Heat, Ansible, and Mistral use YAML for these things... and
> > I
> > think it works well. Understood you have an opinion on this, but I
> > don't share the view that everything works better when written in
> > Python. Take Puppet for example, we interface with that via Hiera.
> > 
> > People will pay attention because we'll be able to add features
> > faster.
> > By not having to build our own API and plumbing we can focus on
> > actual
> > problems rather than boilerplate Python API code.
> > 
> 
> I totally agree with you Dan, and I'd only add that if you can make
> it work with Ansible you gain a huge amount of potential development
> capability in the greater Ansible community.
> 
> > > 
> > > 6) On the testing note, how do we test these workflows?  Do we
> > > know
> > > what
> > > happens when step X fails?  How do we test that they handle it
> > > properly
> > > in an automated and repeatable way?  In Python these are largely
> > > easy
> > > questions to answer: unit tests.  How do you unit test YAML? 
> > 
> > The actions are all unit testable Python.
> > 
> > The workflows themselves would all get tested as part of our CI.
> > With
> > Mistral workflows and the integration I'm proposing with both the
> > CLI
> > and UI we'd have the same API driven workflows tested in both
> > cases. We
> > don't short circuit the API and call into a library like we are
> > doing
> > today for tripleo-common.
> > 
> > >  This is a
> > > big reason I'm not even crazy about having Mistral on the back
> > > end of
> > > a
> > > TripleO API.  We'd be going from code that we can test and prove
> > > works
> > > in a variety of scenarios, to YAML that is tested and proven to
> > > work
> > > in
> > > exactly the three scenarios we run in CI.  This is basically the
> > > same
> > > situation we had with tripleo-incubator, and it was bad there
> > > too.
> > > 
> > > I dunno.  Maybe I'm too late to this party to have any impact on
> > > the
> > > discussion, but I very much do not like the direction we're going
> > > and
> > > I
> > > would be remiss if I didn't at least point out my concerns with
> > > it.
> > 
> > You aren't late to the party. But I would encourage you to look
> > closely
> > at the Mistral demos and examples that have been posted to
> > openstack-
> > dev before commenting further. Try them out, try Ansible (tower),
> > try
> > Mistral, and then come back and have a hard look at what we are
> > trying
> > to do by building our own TripleO API.
> > 
> > To me the crux of the problem isn't that we should expect other
> > projects to build our APIs for us. Rather it is using the right
> > tools
> > for the right jobs. TripleO has gotten off on the wrong path a few
> > times. We tried to roll our own config manage tooling and that
> > didn't
> > work out so well. I hate to see us go down the path of trying to
> > write
> > our own deployment/workflow API when in fact we've already got what
> > exactly what we need in OpenStack already. And a community already
> > exists around it as well...
> > 
> 
> Agree 100%. Note that we did our own config management because we
> didn't
> want to bless one or the other, so it wasn't without a good reason.
> The
> reality is that it was more important that we have a mature solution
> than needing to avoid playing favorites. I think, in this case,
> Mistral
> is in a similar situation. Ansible is a fantastic workflow language,
> with a mature implementation and a very active community. I
> understand
> that Mistral has its own ideas about how that should work, and I
> don't
> want to downplay the hard work that has gone into that. But I truly
> believe that this is a similar situation, and I'd rather not see
> history
> repeated.
> 
> _____________________________________________________________________
> _____
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubs
> cribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev