Open Stack

I don't object to the need of an OpenStack controller and a vm need to communicate. I object to every project coming up with their own solution for that communication channel, when there really is no benefit to them being different. It is one thing amongst many that has made being an Operator's life very difficult as the individual OpenStack projects don't consider all the stuff they are asking an op to do, in aggregate.

This has parallels with, say, college professors. They often do not talk to each other, and will give out a fair amount of homework, which, by itself doesn't sound unreasonable, but when you have a lot of classes, they can add up sometimes to periods of time where you have an unreasonable workload. Its not such a problem in academia, as students are usually kind of a captive audience. Operators can choose to move on though.

Other then the specific messages being sent (payload) between heat and vm, trove and vm, sahara and vm, etc, are the needs of the base communication channel any different between the projects? I don't think there are. But today each projects workarounds/solutions/monitoring are all different.

I feel like I'm just the messenger here. Please don't shoot me. I'm generally trying to provide good, though hard feedback to better a perceived problem. Many others would just stay silent or leave rather then have the difficult discussion.

For reference, have a look here:
https://www.google.com/trends/explore#q=%2Fm%2F0cm87w_&cmpt=q&tz=Etc%2FGMT%2B7

The slope of the line has dramatically changed. I'm sure there are many reasons for it, but the difficulty of being an OpenStack operator or app developer in a world where there are increasing numbers of alternatives I'm sure plays a part. (Some other things, docker and docker COE's staring to gain traction, the introduction of the big tent)

The openness of the big tent has allowed more choice in the ecosystem, which seems good, but I think it has also stalled development on the sorts of cross project issues that were important to our users and are now looking elsewhere for solutions. I have to admit, I've gotten frustrated after years of waiting, and trying to fix things, I'm starting to switch some systems away from OpenStack to things that are easier to deal with. Again, please understand, there is no malice here, just stating the way things are. We should try and fix it so this is not the case.

As for the specific  Trove security risk mentioned below, you can have a zaqar queue per vm, not per tenant, and compromising the vm only compromise control of that one vm. Thats much preferable to the current situation. But again, this is in no way Trove specific. We should be talking this issue through with all the related projects.

Service tenants are not an option for us today, as we have users that buy hardware that gets dedicated to them. they use flavors with permissions to land vm's onto their hardware pools. Service tenants don't have access to those flavors. Again, this is a solution that was thought up and provided by a single project (Trove) but should be considered at the general OpenStack level. It isn't specific to Trove, but affects Sahara, Magnum, etc too. There isn't anything Trove specific about the issue or the solution.

There needs to be a general place... technical committee? that helps design these sorts of solutions that deal with the needs of multiple OpenStack projects and operators and pushes through features (like nova service vm's) that fills in the current implementation gaps to get good solutions. This, everyone for themselves way of solving things isn't working so good.
 
Thanks,
Kevin

________________________________________
From: Amrith Kumar [amrith at tesora.com]
Sent: Saturday, July 16, 2016 8:41 AM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for all)

> -----Original Message-----
> From: Fox, Kevin M [mailto:Kevin.Fox at pnnl.gov]
> Sent: Friday, July 15, 2016 2:37 PM
> To: OpenStack Development Mailing List (not for usage questions)
> <openstack-dev at lists.openstack.org>
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for
> all)
>
> Some specific things:
>
> Magnum trying to not use Barbican as it adds an addition dependency. See
> the discussion on the devel mailing list for details.
>
> Horizon discussions at the summit around wanting to use Zaqar for dynamic
> ui updates instead of polling, but couldn't depend on a non widely
> deployed subsystem.
>
> Each Advanced OpenStack Service implementing a guest controller
> communication channel that are incompatible with each other and work
> around communications issues differently. This makes a lot more pain for
> Ops to debug or architect a viable solution. For example:
>  * Sahara uses ssh from the controllers to the vms. This does not play
> well with tenant networks. They have tried to work around this several
> ways:
>     * require every vm to have a floating ip. (Unnecessary attack surface)
>     * require the controller to be on the one and only network node (Uses
> ip netns exec to tunnel. Doesn't work for large clouds)
>     * Double tunnel ssh via the controller vm's. so some vms have fips,
> some don't. Better then all, but still not good.
>   * Trove uses Rabbit for the guest agent to talk back to the controllers.
> This has traffic going the right direction to work well with tenant
> networks.
>     * But Rabbit is not multitenant so a security risk if any user can get
> into any one of the database vm's.
> Really, I believe the right solution is to use a multitenant aware message
> queue so that the guest agent can pull in the right direction for tenant
> networks, and not have the security risk. We have such a system already,
> Zaqar, but its not widely deployed and projects don't want to depend on
> other projects that aren't widely deployed.

[amrith] Kevin, to the best of my knowledge, Zaqar does not eliminate Trove's security risks if one can compromise a database's VM's. It merely reduces the extent of the compromise (to the tenant whose vm's have been compromised). This understanding is based on a conversation with the Zaqar folks in Vancouver, and later in Tokyo on possible integrations with Trove.

I will observe that it is perfectly possible to operate Trove with a service tenant and thereby eliminate any opportunity for a user to be able to compromise a guest instance that has been created with a rational image (disabled ssh, etc.,). Since the image creation and registration is an operator activity, it places the ability to secure the Trove deployment well within the control of the operator. It is also the case that one can further tighten things up by making each guest instance use its own private RabbitMQ credentials (something that the code currently does not support).

Setting those things aside, let me also point out that if one wants to implement a database as a service, then the controller needs to be able to perform some management of the database instances. One way of doing that is the current architecture that has the guest agent on the guest instance, and another is to have a remote guest agent. One way or another, there is going to have to be communication between the controller and the guest instance. I believe that there is no way around that. I'm unclear why it is unacceptable that the guest instance have a network interface on a controller network with tight firewall protections on what traffic is allowed there.

I believe that each project should be allowed to adopt the solution(s) that are best suited to their own situation, within reason. Mandating that projects shall use a specific solution will, I believe, lead to a worse overall outcome.

>
> The lack of Instance Users has caused lots of projects to try and work
> around the lack thereof. I know for sure, Mangum, Heat, and Trove work
> around the lack. I'm positive others have too. As an operator, it makes me
> have to very carefully consider all the tradeoffs each project made, and
> decide if I can accept the same risk they assumed. Since each is
> different, thats much harder.
>
> I'm sure there are more examples. but I hope you get I'm not just trying
> to troll.
>
> The TC did apply inconsistant rules on letting projects in. That was for
> sure a negative before the big tent. But it also provided a way to apply
> pressure to projects to fix some of the issues that multiple projects
> face, and that plague user/operators and raise the whole community up, and
> that has fallen to the wayside since. Which is a big negative now. Maybe
> that could be bolted on top of the Big Tent I don't know.
>

[amrith] While I often hear that the TC does not make consistent rules, and take strong decisions when it comes to someone else's project, we as a community also seem to not like it when the TC makes a consistent rule or a hard decision that impacts our project.

> I could write a very long description about the state of being an
> OpenStack App developer too that touches on all the problems with getting
> a consistent target and all the cross project communication issues there
> of. But thats probably for some other time.
>
> Thanks,
> Kevin
>
> ________________________________________
> From: Jay Pipes [jaypipes at gmail.com]
> Sent: Friday, July 15, 2016 8:17 AM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [tc][all] Big tent? (Related to Plugins for
> all)
>
> Kevin, can you please be *specific* about your complaints below? Saying
> things like "less project communication" and "projects not working
> together because of fear of adding dependencies" and "worse user
> experience" are your personal opinions. Please back those opinions up
> with specific examples of what you are talking about so that we may
> address specific things and not vague ideas.
>
> Also, the overall goal of the Big Tent, as I've said repeatedly and
> people keep willfully ignoring, was *not* to "make the community more
> inclusive". It was to replace the inconsistently-applied-by-the-TC
> *subjective* criteria for project applications to OpenStack with an
> *objective* list of application requirements that could be
> *consistently* reviewed by the TC.
>
> Thanks,
> -jay
>
> On 07/14/2016 01:30 PM, Fox, Kevin M wrote:
> > I'm going to go ahead and ask the difficult question now as the answer
> is relevant to the attached proposal...
> >
> > Should we reconsider whether the big tent is the right approach going
> forward?
> >
> > There have been some major downsides I think to the Big Tent approach,
> such as:
> >   * Projects not working together because of fear of adding extra
> dependencies Ops don't already have
> >   * Reimplementing features, badly, over and over again in different
> projects instead of standardizing something.
> >   * More projects being created due to politics and not technical
> reasons.
> >   * Less cross project communication
> >   * Greater Operator pain at trying to assemble a more loose
> confederation of projects into something useful.
> >   * General pushing off more and more work to Operators/Users to deal
> with.
> >   * Worse User experience as cross project issues aren't being
> addressed.
> >   * Previously incubated projects such as Nova, Swift, etc getting a
> disproportionate say in things as they have a greater current user base,
> and its increasingly hard now for new projects to gain any traction.
> >   * Much less community pressure on projects to work together to elevate
> everyone. Architectural decisions are now made at individual project level
> with little done at the OpenStack level.
> >   * etc...
> >
> > The overall goal of the Big Tent was to make the community more
> inclusive. That I think has mostly happened. Which is good. But It also
> seems to have fractured the community more into insular islands and made
> the system harder for our operators/users. Which is bad.
> >
> > Maybe the benefits outweigh the drawbacks. I'm not sure. But I think its
> probably time to consider if it has been a net positive and should be
> further refined to try and address some of these problems, or a net
> negative and different approaches be explored.
> >
> > Thanks,
> > Kevin
> > ________________________________________
> > From: Hayes, Graham [graham.hayes at hpe.com]
> > Sent: Thursday, July 14, 2016 12:21 PM
> > To: OpenStack Development Mailing List (not for usage questions);
> openstack-tc at lists.openstack.org
> > Subject: [openstack-dev] [tc][all] Plugins for all
> >
> > I just proposed a review to openstack/governance repo [0] that aims
> > to have everything across OpenStack be plugin based for all cross
> > project interaction, or allow all projects access to the same internal
> > APIs and I wanted to give a bit of background on my motivation, and how
> > it came about.
> >
> > Coming from a smaller project, I can see issues for new projects,
> > smaller projects, and projects that may not be seen as "important".
> >
> > As a smaller project trying to fit into cross project initiatives,
> > (and yes, make sure our software looks at least OK in the
> > Project Navigator) the process can be difficult.
> >
> > A lot of projects / repositories have plugin interfaces, but also
> > have project integrations in tree, that do not follow the plugin
> > interface. This makes it difficult to see what a plugin can, and
> > should do.
> >
> > When we moved to the big tent, we wanted as a community to move to
> > a flatter model, removing the old integrated status.
> >
> > Unfortunately we still have areas when some projects are more equal -
> > there is a lingering set of projects who were integrated at the point
> > in time that we moved, and have preferential status.
> >
> > A lot of the effects are hard to see, and are not insurmountable, but
> > do cause projects to re-invent the wheel.
> >
> > For example, quotas - there is no way for a project that is not nova,
> > neutron, cinder to hook into the standard CLI, or UI for setting
> > quotas. They can be done as either extra commands
> > (openstack dns quota set --foo bar) or as custom panels, but not
> > the way other quotas get set.
> >
> > Tempest plugins are another example. Approximately 30 of the 36
> > current plugins are using resources that are not supposed to be
> > used, and are an unstable interface. Projects in tree in tempest
> > are at a much better position, as any change to the internal
> > API will have to be fixed before the gate merges, but other
> > out of tree plugins are in a place where they can be broken at any
> > point.
> >
> > None of this is meant to single out projects, or teams. A lot
> > of the projects that are in this situation have inordinate amounts of
> > work placed on them by the big-tent, and I can emphasize with why things
> > are this way. These were the examples that currently stick out
> > in my mind, and I think we have come to a point where we need to make
> > a change as a community.
> >
> > By moving to a "plugins for all" model, these issues are reduced.
> > It undoubtedly will cause more, but it is closer to our goal
> > of Recognizing all our community is part of OpenStack, and
> > differentiate projects by tags.
> >
> > This won't be a change that happens tomorrow, next week, or even next
> > cycle, but think as a goal, we should start moving in this direction
> > as soon as we can, and start building momentum.
> >
> > Thanks,
> >
> > Graham
> >
> > 0 - https://review.openstack.org/342366
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev