Open Stack

TL;DR: From ~600 nova specific config options are:
            ~140 at a central location with an improved help text
            ~220 options in open reviews (currently on hold)
            ~240 options todo


Background
==========
Nova has a lot of config options. Most of them weren't well
documented and without looking in the code you probably don't
understand what they do. That's fine for us developers but the ops
had more problems with the interface we provide for them [1]. After
the Mitaka summit we came to the conclusion that this should be 
improved, which is currently in progress with blueprint [2].


Current Status
==============
After asking on the ML for help [3] the progress improved a lot. 
The goal is clear now and we know how to achieve it. The organization 
is done via [4] which also has a section of "odd config options". 
This section is important for a later step when we want do deprecate 
config options to get rid of unnecessary ones. 

As we reached the Mitaka-3 milestone we decided to put the effort [5] 
on hold to stabilize the project and focus the review effort on bug 
fixes. When the Newton cycle opens, we can continue the work. The 
current result can be seen in the sample "nova.conf" file generated 
after each commit [6]. The appendix at the end of this post shows an
example.

All options we have will be treated that way and moved to a central
location at "nova/conf/". That's the central location which hosts
now the interface to the ops. It's easier to get an overview now.
The appendix shows how the config options were spread at the beginning
and how they are located now.

I initially thought that we have around 800 config options in Nova
but I learned meanwhile that we import a lot from other libs, for 
example from "oslo.db" and expose them as Nova options. We have around
600 Nova specific config options, and ~140 are already treaded like
described above and ca. 220 are in the pipeline of open reviews.
Which leaves us ~240 which are not looked at yet.


Outlook
=======
The numbers of the beginning of this ML post make me believe that we
can finish the work in the upcoming Newton cycle. "Finished" means
here: 
* all config options we provide to our ops have proper and usable docs
* we have an understanding which options don't make sense anymore
* we know which options should get stronger validation to reduce errors

I'm looking forward to it :)


Thanks
======
I'd like to thank all the people who are working on this and making
this possible. A special thanks goes to Ed Leafe, Esra Celik and
Stephen Finucane. They put a tremendous amount of work in it.


References:
===========
[1] 
http://lists.openstack.org/pipermail/openstack-operators/2016-January/009301.html
[2] https://blueprints.launchpad.net/nova/+spec/centralize-config-options
[3] 
http://lists.openstack.org/pipermail/openstack-dev/2015-December/081271.html
[4] https://etherpad.openstack.org/p/config-options
[5] Gerrit reviews for this topic: 
https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/centralize-config-options
[6] The sample config file which gets generated after each commit:
    http://docs.openstack.org/developer/nova/sample_config.html


Appendix
========

Example of the help text improvement
-----------------------------------
As an example, compare the previous documentation of the scheduler 
option "scheduler_tracks_instance_changes". 
Before we started:

    # Determines if the Scheduler tracks changes to instances to help 
    # with its filtering decisions. (boolean value)
    #scheduler_tracks_instance_changes = true

After the improvement:

    # The scheduler may need information about the instances on a host 
    # in order to evaluate its filters and weighers. The most common 
    # need for this information is for the (anti-)affinity filters, 
    # which need to choose a host based on the instances already running
    # on a host.
    #
    # If the configured filters and weighers do not need this information,
    # disabling this option will improve performance. It may also be 
    # disabled when the tracking overhead proves too heavy, although 
    # this will cause classes requiring host usage data to query the 
    # database on each request instead.
    #
    # This option is only used by the FilterScheduler and its subclasses;
    # if you use a different scheduler, this option has no effect.
    #
    # * Services that use this:
    #
    #     ``nova-scheduler``
    #
    # * Related options:
    #
    #     None
    #  (boolean value)
    #scheduler_tracks_instance_changes = true


The spread of config options in the tree
----------------------------------------
We started with this in November 2015. It's the Nova project tree and 
the numbers behind the package name are the numbers of config options
declared in that package (config options declared in sub-packages are
not accumulated).

    Based on:
    commit 201090b0bcb 
    Date: Thu Nov 19 04:08:51 2015 +0000

     nova [87]
    --- CA [0]
    ------ newcerts [0]
    ------ private [0]
    ------ projects [0]
    ------ reqs [0]
    --- api [3]
    ------ ec2 [14]
    ------ metadata [6]
    ------ openstack [6]
    --------- compute [2]
    ------------ legacy_v2 [2]
    --------------- contrib [6]
    ------------ schemas [0]
    ------------ views [0]
    ------ validation [0]
    --- cells [25]
    ------ filters [0]
    ------ weights [3]
    --- cert [2]
    --- cloudpipe [6]
    --- cmd [15]
    --- compute [49]
    ------ monitors [2]
    --------- cpu [0]
    ------ resources [0]
    --- conductor [5]
    ------ tasks [1]
    --- conf [0]
    --- console [16]
    --- consoleauth [3]
    --- db [4]
    ------ sqlalchemy [13]
    --------- api_migrations [0]
    ------------ migrate_repo [0]
    --------------- versions [0]
    --------- migrate_repo [0]
    ------------ versions [0]
    --- hacking [0]
    --- image [14]
    ------ download [3]
    --- ipv6 [1]
    --- keymgr [5]
    --- mks [2]
    --- network [68]
    ------ neutronv2 [14]
    ------ security_group [1]
    --- objects [2]
    --- objectstore [3]
    --- openstack [0]
    ------ common [4]
    --- pci [2]
    --- rdp [2]
    --- scheduler [14]
    ------ client [0]
    ------ filters [15]
    ------ weights [6]
    --- servicegroup [1]
    ------ drivers [4]
    --- spice [6]
    --- virt [18]
    ------ disk [2]
    --------- mount [1]
    --------- vfs [1]
    ------ hyperv [16]
    ------ image [0]
    ------ ironic [10]
    ------ libvirt [45]
    --------- storage [2]
    --------- volume [19]
    ------ vmwareapi [24]
    ------ xenapi [36]
    --------- client [2]
    --------- image [8]
    --- vnc [8]
    --- volume [8]
    ------ encryptors [0]
    --- wsgi [0]
    ----------------------------
    Number of total options: 637


As of now, we have these numbers (please note the "nova/conf/" folder)

    Based on:
    commit 040df06bfc4 
    Date: Wed Mar 2 11:09:24 2016 +0000

     nova [72]
    --- CA [0]
    ------ newcerts [0]
    ------ private [0]
    ------ projects [0]
    ------ reqs [0]
    --- api [3]
    ------ ec2 [0]
    ------ metadata [6]
    ------ openstack [7]
    --------- compute [2]
    ------------ legacy_v2 [2]
    --------------- contrib [6]
    ------------ schemas [0]
    ------------ views [0]
    ------ validation [0]
    --- cells [0]
    ------ filters [0]
    ------ weights [0]
    --- cert [0]
    --- cloudpipe [6]
    --- cmd [11]
    --- compute [41]
    ------ monitors [2]
    --------- cpu [0]
    ------ resources [0]
    --- conductor [1]
    ------ tasks [1]
    --- conf [136]
    --- console [11]
    --- consoleauth [3]
    --- db [4]
    ------ sqlalchemy [13]
    --------- api_migrations [0]
    ------------ migrate_repo [0]
    --------------- versions [0]
    --------- migrate_repo [0]
    ------------ versions [0]
    --- hacking [0]
    --- image [15]
    ------ download [3]
    --- ipv6 [1]
    --- keymgr [5]
    --- mks [2]
    --- network [68]
    ------ neutronv2 [7]
    ------ security_group [1]
    --- objects [2]
    --- openstack [0]
    ------ common [0]
    --- pci [0]
    --- rdp [2]
    --- scheduler [0]
    ------ client [0]
    ------ filters [0]
    ------ weights [0]
    --- servicegroup [1]
    ------ drivers [0]
    --- spice [6]
    --- virt [7]
    ------ disk [0]
    --------- mount [0]
    --------- vfs [1]
    ------ hyperv [15]
    ------ image [0]
    ------ ironic [0]
    ------ libvirt [48]
    --------- storage [2]
    --------- volume [19]
    ------ vmwareapi [24]
    ------ xenapi [36]
    --------- client [2]
    --------- image [8]
    --- vnc [0]
    --- volume [8]
    ------ encryptors [0]
    --- wsgi [0]
    ----------------------------
    Number of total options: 610


Regards, Markus Zoeller (markus_z)