Open Stack

HTML render: http://www.openstack.org/blog/2016/06/openstack-developer-mailing-list-digest-20160617/

SuccessBot Says
===============
* Qiming: Senlin has completed API migration from WADL.
* Mugsie: Kiall Fixed the gate - development can now continue!!!
* notmyname: exactly 6 years ago today, Swift was put into production
* kiall: DNS API reference is live [1].
* sdague: Nova legacy v2 api code removed [2].
* HenryG: Last remnant of oslo incubator removed from Neutron [3].
* dstanek: I was able to perform a roundtrip between keystone and testshib.org
  using my new SAML2 middleware!
* Sdague: Nova now defaults to Glance v2 for image operations [4].
* Ajaeger: First Project Specific Install Guide is published - congrats to the
  heat team!
* Jeblair: There is no Jenkins, only Zuul.
* All: https://wiki.openstack.org/wiki/Successes


Require A Level Playing Field for OpenStack Projects
====================================================
* Thierry Carrez proposes a new requirement [5] for OpenStack “official”
  projects.
* An important characteristic of open collaboration grounds is they need to be
  a level playing field. No specific organization can be given an unfair
  advantage.

  - Projects that are blessed as “official” project teams need to operate in
    a fair manner. Otherwise they could be essentially a trojan horse for
    a given organization.
  - If in a given project, developers from one specific organization benefit
    from access specific knowledge or hardware, then the project should be
    rejected under the “open community” rule.
  - Projects like Cinder provide an interesting grey area, but as long as all
    drivers are in and there is a fully functional (and popular) open source
    implementation there is likely no specific organization considered as
    unfairly benefiting.

* Neutron plugin targeting a specific piece of networking hardware would likely
  given an unfair advantage to developers of the hardware's manufacturer
  (having access to hardware for testing and being able to see and make changes
  to its proprietary source code).
* Open source projects that don't meet the open community requirement can still
  exist in the ecosystem (developed using gerrit and openstack/* git
  repository, gate testing, but as an unofficial project.
* Full thread: http://lists.openstack.org/pipermail/openstack-dev/2016-June/thread.html#97307


Add Option to Disable Some Strict Response Checking for Interop Testing
========================================================================
* Nova introduced their API micro version change [6]
* QA team adds strict API schema checking to Tempest to ensure no additional
  Nova API responses [7][8].
  - In the last year, three vendors participating in the OpenStack powered
    trademark program were impacted by this [9].
* DefCore working group determines guidelines for the OpenStack powered
  program.

  - Includes capabilities with associated functional tests from Tempest that
    must pass.
  - There is a balance of future direction of development with lagging
    indicators of deployments and user adoption.

* A member of the working group Chris Hoge would like to implement a temporary
  waiver for strict API checking requirements.

  - While this was discussed publicly in the developer community and took some
    time to implement. It still landed quickly, and broke several existing
    deployments overnight.
  - It's not viable for downstream deployers to use older versions of Tempest
    that don't have these strict response checking, due to the TC resolution
    passed [10] to advise DefCore to use Tempest as the single source of

    capability testing.
* Proposal:
  - Short term:
    + There will be a blueprint and patch to Tempest that allows configuration
      of a grey-list of Nova APIs which strict response checking on additional
      properties will be disabled.
    + Using this code will emit a deprecation warning.
    + This will be removed 2017.01.
    + Vendors are required to submit the grey-list of APIs with additional
      response data that would be published to their marketplace entry.

  - Long term:
    + Vendors will be expected to work with upstream to update the API
      returning additional data.
    + The waiver would no longer be allowed after the release of 2017.01
    guidleine.

* Former QA PTL Matthew Treinish feels this a big step backwards.
  - Vendors who have implemented out of band extensions or injected additional
    things in responses believe that by doing so they're interoperable. The API
    is not a place for vendor differentation.
  - Being a user of several clouds, random data in the response makes it more
    difficult to write code against. Which are the vendor specific responses?
* Alternatives to not giving vendors more time in the market:
  - Having some vendors leave the the Powered program unnecessarily weakening
    it.
  - Force DefCore to adopt non-upstream testing, either as a fork or an
    independent test suite.
* If the new enforcement policies had been applied by adding new tests to
  Tempest, then DefCore could have added them using it's processes over
  a period of time.
  - Instead behavior to a bunch of existing tests changed.
* Tempest master today supports all currently supported stable branches.
  - Tags are made in the git repository support for a release is added/dropped.
    + Branchless Tempest was originally started back in Icehouse release and
      was implemented to enforce the API is the same across release boundaries.
* If DefCore wants the lowest common denominator for Kilo, Liberty, and Mitaka
  there's a tag for that [11]. For Juno, Kilo, Liberty the tag would be [12].
* Full thread: http://lists.openstack.org/pipermail/openstack-dev/2016-June/thread.html#97349


There Is No Jenkins, Only Zuul
==============================
* Since the inception of OpenStack, we have used Jenkins to perform testing and
  artifact building.
  - When we only had two git repositories, we have one Jenkins master and a few
    slaves. This was easy to maintain.
  - Things have grown significantly with 1,200 git repositories, 8,000 jobs
    spread across 8 Jenkins masters and 800 dynamic slave nodes.
* Jenkins job builder [13] was created to create 8,000 jobs from a templated
  YAML.
* Zuul [14] was created to drive project automation in directing our testing,
  running tens of thousands of jobs each day. Responding to:
  - Code reviews
  - Stacking potential changes to be tested together.
* Zuul version 3 has major changes:
  - Easier to run jobs in multi-node environments
  - Easy to manage large number of jobs
  - Job variations
  - Support in-tree job configuration
  - Ability to define jobs using Ansible
* While version 3 is still in development, it's today capable of running our
  jobs entirely.
* As of June 16th, we have turned off our last Jenkins master and all of our
  automation is being run by Zuul.
  - Jenkins job builder has contributors beyond OpenStack, and will be
    continued to be maintained by them.
* Full thread: http://lists.openstack.org/pipermail/openstack-dev/2016-June/thread.html#97584


Languages vs. Scope of “OpenStack”
==================================
* Where does OpenStack stop, and where does the wider open source community
  start? Two options:
  - If OpenStack is purely an “integration engine” to lower-level technologies
    (e.g. hypervisors, databases, block storage) the scope is limited and
    Python should be plenty and we don't need to fragment our community.
  - If OpenStack is “whatever it takes to reach our mission”, then yes we need
    to add one language to cover lower-level/native optimization.
* Swift PTL John Dickinson mentions defining the scope of OpenStack projects
  does not define the languages needed to implement them. The considerations
  are orthogonal.
  - OpenStack is defined as whatever is takes to fulfill the mission statement.
  - Defining “lower level” is very hard. Since the Nova API is listening to
    public network interfaces and coordinating with various services in
    a cluster, it lower level enough to consider optimizations.
    + There are a large number of open source projects and libraries that
      OpenStack needs to fulfill its mission that are not “OpenStack”: Python,
      MySQL, KVM, Ceph, OpenvSwitch.
* Another approach is product-centric. “Lower-level pieces are OpenStack
  dependencies, rather that OpenStack itself.”
  - Not governed by the TC, and it can use any language and tool deemed
    necessary.
* Do we want to be in the business of building data plane services that will
  run into Python limitations?
  - Control plane services are very unlikely to ever hit a scaling concern
    where rewriting in another language is needed.
* Swift hit it first because of the maturity of the project and they are now
  focused on this kind of optimization.
  - Glance (partially data plane) did hit this limit and mitigated by folks
    using Ceph and exposing that directly to Nova. So now Glance only cares
    about location and metadata. Dependencies like Ceph care about data plane.
* The resolution for the Go programming language was discussed in previous
  Technical Committee meetings and was not passed [14]. John Dickinson and
  others do plan to carry another effort forward for Swift to have an exception
  for usage of the language.
* Full thread: http://lists.openstack.org/pipermail/openstack-dev/2016-May/thread.html#95452


[1] - http://developer.openstack.org/api-ref/dns/
[2] - https://review.openstack.org/324466
[3] - https://review.openstack.org/290305
[4] - https://review.openstack.org/#/c/321551/
[5] - https://review.openstack.org/#/c/329448/
[6] - https://specs.openstack.org/openstack/nova-specs/specs/kilo/implemented/api-microversions.html
[7] - http://lists.openstack.org/pipermail/openstack-dev/2015-February/057613.html
[8] - https://review.openstack.org/#/c/156130
[9] - http://lists.openstack.org/pipermail/defcore-committee/2016-January/000986.html
[10] - http://git.openstack.org/cgit/openstack/governance/tree/resolutions/20160504-defcore-test-location.rst
[11] - http://git.openstack.org/cgit/openstack/tempest/tag/?h=12.0.0
[12] - http://git.openstack.org/cgit/openstack/tempest/tag/?h=8
[13] - http://docs.openstack.org/infra/jenkins-job-builder/
[14] - https://review.openstack.org/312267