[openstack-dev] [magnum] Discussion of supporting single/multiple OS distro

Kai Qiang Wu wkqwu at cn.ibm.com
Wed Mar 16 01:20:46 UTC 2016

Hi  Stdake,

There is a patch about Atomic 23 support in Magnum.  And atomic 23 uses
kubernetes 1.0.6, and docker 1.9.1.
>From Steve Gordon, I learnt they did have a two-weekly release. To me it
seems each atomic 23 release not much difference, (minor change)
The major rebases/updates may still have to wait for e.g. Fedora Atomic 24.

So maybe we not need to test every Atomic 23 two-weekly.
Pick one or update old, when we find it is integrated with new kubernetes
or docker, etcd etc. If other small changes(not include security), seems
not need to update so frequently, it can save some efforts.

What do you think ?


Best Wishes,
Kai Qiang Wu (吴开强  Kennan)
IBM China System and Technology Lab, Beijing

E-mail: wkqwu at cn.ibm.com
Tel: 86-10-82451647
Address: Building 28(Ring Building), ZhongGuanCun Software Park,
         No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China
Follow your heart. You are miracle!

From:	"Steven Dake (stdake)" <stdake at cisco.com>
To:	"OpenStack Development Mailing List (not for usage questions)"
            <openstack-dev at lists.openstack.org>
Date:	16/03/2016 03:23 am
Subject:	Re: [openstack-dev] [magnum] Discussion of supporting
            single/multiple OS distro

WFM as long as we stick to the spirit of the proposal and don't end up in a
situation where there is only one distribution.  Others in the thread had
indicated there would be only one distribution in tree, which I'd find
disturbing for reasons already described on this thread.

While we are about it, we should move to the latest version of atomic and
chase atomic every two weeks on their release.  Thoughts?


From: Hongbin Lu <hongbin.lu at huawei.com>
Reply-To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev at lists.openstack.org>
Date: Monday, March 14, 2016 at 8:10 PM
To: "OpenStack Development Mailing List (not for usage questions)" <
openstack-dev at lists.openstack.org>
Subject: Re: [openstack-dev] [magnum] Discussion of supporting
single/multiple OS distro

            From: Adrian Otto [mailto:adrian.otto at rackspace.com]
            Sent: March-14-16 4:49 PM
            To: OpenStack Development Mailing List (not for usage
            Subject: Re: [openstack-dev] [magnum] Discussion of supporting
            single/multiple OS distro


            I think you may have misunderstood our intent here. We are not
            seeking to lock in to a single OS vendor. Each COE driver can
            have a different OS. We can have multiple drivers per COE. The
            point is that drivers should be simple, and therefore should
            support one Bay node OS each. That would mean taking what we
            have today in our Kubernetes Bay type implementation and
            breaking it down into two drivers: one for CoreOS and another
            for Fedora/Atomic. New drivers would start out in a contrib
            directory where complete functional testing would not be
            required. In order to graduate one out of contrib and into the
            realm of support of the Magnum dev team, it would need to have
            a full set of tests, and someone actively maintaining it.
      OK. It sounds like the proposal allows more than one OS to be
      in-tree, as long as the second OS goes through an incubation process.
      If that is what you mean, it sounds reasonable to me.

            Multi-personality driers would be relatively complex. That
            approach would slow down COE specific feature development, and
            complicate maintenance that is needed as new versions of the
            dependency chain are bundled in (docker, k8s, etcd, etc.). We
            have all agreed that having integration points that allow for
            alternate OS selection is still our direction. This follows the
            pattern that we set previously when deciding what networking
            options to support. We will have one that’s included as a
            default, and a way to plug in alternates.

            Here is what I expect to see when COE drivers are implemented:

            Docker Swarm:
            Default driver Fedora/Atomic
            Alternate driver: TBD

            Default driver Fedora/Atomic
            Alternate driver: CoreOS

            Apache Mesos/Marathon:
            Default driver: Ubuntu
            Alternate driver: TBD

            We can allow an arbitrary number of alternates. Those TBD items
            can be initially added to the contrib directory, and with the
            right level of community support can be advanced to defaults if
            shown to work better, be more straightforward to maintain, be
            more secure, or whatever criteria is important to us when
            presented with the choice. Such criteria will be subject to
            community consensus. This should allow for free experimentation
            with alternates to allow for innovation. See how this is not
            locking in a single OS vendor?


                  On Mar 14, 2016, at 12:41 PM, Steven Dake (stdake) <
                  stdake at cisco.com> wrote:


                  When we are at a disagreement in the Kolla core team, we
                  have the Kolla core reviewers vote on the matter. This is
                  typical standard OpenStack best practice.

                  I think the vote would be something like
                  "Implement one OS/COE/network/storage prototype, or
                  implement many."

                  I don't have a horse in this race, but I think it would
                  be seriously damaging to Magnum to lock in to a single


                  From: Hongbin Lu <hongbin.lu at huawei.com>
                  Reply-To: "OpenStack Development Mailing List (not for
                  usage questions)" <openstack-dev at lists.openstack.org>
                  Date: Monday, March 7, 2016 at 10:06 AM
                  To: "OpenStack Development Mailing List (not for usage
                  questions)" <openstack-dev at lists.openstack.org>
                  Subject: Re: [openstack-dev] [magnum] Discussion of
                  supporting single/multiple OS distro

                         From: Corey O'Brien [mailto:coreypobrien at gmail.com
                         Sent: March-07-16 8:11 AM
                         To: OpenStack Development Mailing List (not for
                         usage questions)
                         Subject: Re: [openstack-dev] [magnum] Discussion
                         of supporting single/multiple OS distro

                         Hongbin, I think the offer to support different OS
                         options is a perfect example both of what we want
                         and what we don't want. We definitely want to
                         allow for someone like yourself to maintain
                         templates for whatever OS they want and to have
                         that option be easily integrated in to a Magnum
                         deployment. However, when developing features or
                         bug fixes, we can't wait for you to have time to
                         add it for whatever OS you are promising to
                   It might be true that supporting additional OS could
                   slow down the development speed, but the key question is
                   how much the impact will be. Does it outweigh the
                   benefits? IMO, the impact doesn’t seem to be
                   significant, given the fact that most features and bug
                   fixes are OS agnostic. Also, keep in mind that every
                   features we introduced (variety of COEs, variety of Nova
                   virt-driver, variety of network driver, variety of
                   volume driver, variety of …) incurs a maintenance
                   overhead. If you want an optimal development speed, we
                   will be limited to support a single COE/virt
                   driver/network driver/volume driver. I guess that is not
                   the direction we like to be?

                         Instead, we would all be forced to develop the
                         feature for that OS as well. If every member of
                         the team had a special OS like that we'd all have
                         to maintain all of them.
                   To be clear, I don’t have a special OS, I guess neither
                   do others who disagreed in this thread.

                         Alternatively, what was agreed on by most at the
                         midcycle was that if someone like yourself wanted
                         to support a specific OS option, we would have an
                         easy place for those contributions to go without
                         impacting the rest of the team. The team as a
                         whole would agree to develop all features for at
                         least the reference OS.
                   Could we re-confirm that this is a team agreement? There
                   is no harm to re-confirm it in the design summit/ML/team
                   meeting. Frankly, it doesn’t seem to be.

                         Then individuals or companies who are passionate
                         about an alternative OS can develop the features
                         for that OS.


                         On Sat, Mar 5, 2016 at 12:30 AM Hongbin Lu <
                         hongbin.lu at huawei.com> wrote:

                               From: Adrian Otto [mailto:
                               adrian.otto at rackspace.com]
                               Sent: March-04-16 6:31 PM

                               To: OpenStack Development Mailing List (not
                               for usage questions)
                               Subject: Re: [openstack-dev] [magnum]
                               Discussion of supporting single/multiple OS


                                     On Mar 4, 2016, at 2:41 PM, Steven
                                     Dake (stdake) <stdake at cisco.com>

                                     From: Adrian Otto <
                                     adrian.otto at rackspace.com>
                                     Reply-To: "OpenStack Development
                                     Mailing List (not for usage
                                     questions)" <
                                     openstack-dev at lists.openstack.org>
                                     Date: Friday, March 4, 2016 at 12:48
                                     To: "OpenStack Development Mailing
                                     List (not for usage questions)" <
                                     openstack-dev at lists.openstack.org>
                                     Subject: Re: [openstack-dev] [magnum]
                                     Discussion of supporting
                                     single/multiple OS distro


                                      To be clear, this pursuit is not
                                      about what OS options cloud operators
                                      can select. We will be offering a
                                      method of choice. It has to do with
                                      what we plan to build comprehensive
                                      testing for,
                                This is easy. Once we build comprehensive
                                tests for the first OS, just re-run it for
                                other OS(s).

                                      and the implications that has on our
                                      pace of feature development. My
                                      guidance here is that we resist the
                                      temptation to create a system with
                                      more permutations than we can
                                      possibly support. The relation
                                      between bay node OS, Heat Template,
                                      Heat Template parameters, COE, and
                                      COE dependencies (could-init, docker,
                                      flannel, etcd, etc.) are
                                      multiplicative in nature. From the
                                      mid cycle, it was clear to me that:

                                      1) We want to test at least one OS
                                      per COE from end-to-end with
                                      comprehensive functional tests.
                                      2) We want to offer clear and precise
                                      integration points to allow cloud
                                      operators to substitute their own OS
                                      in place of whatever one is the
                                      default for the given COE.

                                     A COE shouldn’t have a default
                                     necessarily that locks out other
                                     defaults.  Magnum devs are the experts
                                     in how these systems operate, and as
                                     such need to take on the
                                     responsibility of the implementation
                                     for multi-os support.

                                      3) We want to control the total
                                      number of configuration permutations
                                      to simplify our efforts as a project.
                                      We agreed that gate testing all
                                      possible permutations is intractable.

                                     I disagree with this point, but I
                                     don't have the bandwidth available to
                                     prove it ;)

                               That’s exactly my point. It takes a chunk of
                               human bandwidth to carry that
                               responsibility. If we had a system engineer
                               assigned from each of the various upstream
                               OS distros working with Magnum, this would
                               not be a big deal. Expecting our current
                               contributors to support a variety of OS
                               variants is not realistic.
                         You have my promise to support an additional OS
                         for 1 or 2 popular COEs.

                               Change velocity among all the components we
                               rely on has been very high. We see some of
                               our best contributors frequently sidetracked
                               in the details of the distros releasing
                               versions of code that won’t work with ours.
                               We want to upgrade a component to add a new
                               feature, but struggle to because the new
                               release of the distro that offers that
                               component is otherwise incompatible.
                               Multiply this by more distros, and we expect
                               a real problem.
                         At Magnum upstream, the overhead doesn’t seem to
                         come from the OS. Perhaps, that is specific to
                         your downstream?

                               There is no harm if you have 30 gates
                               running the various combinations.
                               Infrastructure can handle the load.  Whether
                               devs have the cycles to make a fully
                               bulletproof gate is the question I think you
                               answered with the word intractable.

                               Actually, our existing gate tests are really
                               stressing out our CI infra. At least one of
                               the new infrastructure providers that
                               replaced HP have equipment that runs
                               considerably slower. For example, our swam
                               functional gate now frequently fails because
                               it can’t finish before the allowed time
                               limit of 2 hours where it could finish
                               substantially faster before. If we expanded
                               the workload considerably, we might quickly
                               work to the detriment of other projects by
                               perpetually clogging the CI pipelines. We
                               want to be a good citizen of the openstack
                               CI community. Testing configuration of third
                               party software should be done with third
                               party CI setups. That’s one of the reasons
                               those exist. Ideally, each would be
                               maintained by those who have a strategic
                               (commercial?) interest in support for that
                               particular OS.

                                     I can tell you in Kolla we spend a lot
                                     of cycles just getting basic gating
                                     going of building containers and then
                                     deploying them.  We have even made
                                     inroads into testing the deployment.
                                     We do CentOS, Ubuntu, and soon Oracle
                                     Linux, for both source and binary and
                                     build and deploy.  Lots of gates and
                                     if they aren't green we know the patch
                                     is wrong.

                               Remember that COE’s are tested on nova
                               instances within heat stacks. Starting lots
                               of nova instances within devstack in the
                               gates is problematic. We are looking into
                               using a libvirt-lxc instance type from nova
                               instead of a libvirt-kvm instance to help
                               alleviate this. Until then, limiting the
                               scope of our gate tests is appropriate. We
                               will continue our efforts to make them
                               reasonably efficient.




                                      Note that it will take a thoughtful
                                      approach (subject to discussion) to
                                      balance these interests. Please take
                                      a moment to review the interest
                                      above. Do you or others disagree with
                                      these? If so, why?


                                            On Mar 4, 2016, at 9:09 AM,
                                            Hongbin Lu <
                                            hongbin.lu at huawei.com> wrote:

                                            I don’t think there is any
                                            consensus on supporting single
                                            distro. There are multiple
                                            disagreements on this thread,
                                            including several senior team
                                            members and a project
                                            co-founder. This topic should
                                            be re-discussed (possibly at
                                            the design summit).

                                            Best regards,

                                            From: Corey O'Brien [
                                            mailto:coreypobrien at gmail.com]
                                            Sent: March-04-16 11:37 AM
                                            To: OpenStack Development
                                            Mailing List (not for usage
                                            Subject: Re: [openstack-dev]
                                            [magnum] Discussion of
                                            supporting single/multiple OS

                                            I don't think anyone is saying
                                            that code should somehow block
                                            support for multiple distros.
                                            The discussion at midcycle was
                                            about what the we should gate
                                            on and ensure feature parity
                                            for as a team. Ideally, we'd
                                            like to get support for every
                                            distro, I think, but no one
                                            wants to have that many gates.
                                            Instead, the consensus at the
                                            midcycle was to have 1
                                            reference distro for each COE,
                                            gate on those and develop
                                            features there, and then have
                                            any other distros be maintained
                                            by those in the community that
                                            are passionate about them.

                                            The issue also isn't about how
                                            difficult or not it is. The
                                            problem we want to avoid is
                                            spending precious time
                                            guaranteeing that new features
                                            and bug fixes make it through
                                            multiple distros.


                                            On Fri, Mar 4, 2016 at 11:18 AM
                                            Steven Dake (stdake) <
                                            stdake at cisco.com> wrote:
                                             My position on this is simple.

                                             Operators are used to using
                                             specific distros because that
                                             is what they used in the
                                             90s,and the 00s, and the 10s.
                                             Yes, 25 years of using a
                                             distro, and you learn it
                                             inside and out.  This means
                                             you don't want to relearn a
                                             new distro, especially if your
                                             an RPM user going to DEB or a
                                             DEB user going to RPM.  These
                                             are non-starter options for
                                             operators, and as a result,
                                             mean that distro choice is a
                                             must.  Since CoreOS is a new
                                             OS in the marketplace, it may
                                             make sense to consider placing
                                             it in "third" position in
                                             terms of support.

                                             Besides that problem, various
                                             distribution companies will
                                             only support distros running
                                             in Vms if it matches the host
                                             kernel, which makes total
                                             sense to me.  This means on an
                                             Ubuntu host if I want support
                                             I need to run Ubuntu vms, on a
                                             RHEL host I want to run RHEL
                                             vms, because, hey, I want my
                                             issues supported.

                                             For these reasons and these
                                             reasons alone, there is no
                                             good rationale to remove
                                             multi-distro support  from
                                             Magnum.  All I've heard in
                                             this thread so far is "its too
                                             hard".  Its not too hard,
                                             especially with Heat
                                             conditionals making their way
                                             into Mitaka.


                                             From: Hongbin Lu <
                                             hongbin.lu at huawei.com>
                                             Reply-To: "
                                             openstack-dev at lists.openstack.org
                                             " <
                                             openstack-dev at lists.openstack.org
                                             Date: Monday, February 29,
                                             2016 at 9:40 AM
                                             To: "
                                             openstack-dev at lists.openstack.org
                                             " <
                                             openstack-dev at lists.openstack.org
                                             Subject: [openstack-dev]
                                             [magnum] Discussion of
                                             supporting single/multiple OS

                                             Hi team,

                                             This is a continued discussion
                                             from a review [1]. Corey
                                             O'Brien suggested to have
                                             Magnum support a single OS
                                             distro (Atomic). I disagreed.
                                             I think we should bring the
                                             discussion to here to get
                                             broader set of inputs.

                                             Corey O'Brien
                                             From the midcycle, we decided
                                             we weren't going to continue
                                             to support 2 different
                                             versions of the k8s template.
                                             Instead, we were going to
                                             maintain the Fedora Atomic
                                             version of k8s and remove the
                                             coreos templates from the
                                             tree. I don't think we should
                                             continue to develop features
                                             for coreos k8s if that is
                                             In addition, I don't think we
                                             should break the coreos
                                             template by adding the trust
                                             token as a heat parameter.

                                             Hongbin Lu
                                             I was on the midcycle and I
                                             don't remember any decision to
                                             remove CoreOS support. Why you
                                             want to remove CoreOS
                                             templates from the tree.
                                             Please note that this is a
                                             very big decision and please
                                             discuss it with the team
                                             thoughtfully and make sure
                                             everyone agree.

                                             Corey O'Brien
                                             Removing the coreos templates
                                             was a part of the COE drivers
                                             decision. Since each COE
                                             driver will only support 1
                                             distro+version+coe we
                                             discussed which ones to
                                             support in tree. The decision
                                             was that instead of trying to
                                             support every distro and every
                                             version for every coe, the
                                             magnum tree would only have
                                             support for 1 version of 1
                                             distro for each of the 3 COEs
                                             (swarm/docker/mesos). Since we
                                             already are going to support
                                             Atomic for swarm, removing
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160316/1a03add8/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160316/1a03add8/attachment.gif>

More information about the OpenStack-dev mailing list