Open Stack

On 19 June 2017 at 06:05, Doug Hellmann <doug at doughellmann.com> wrote:
> Excerpts from Michał Jastrzębski's message of 2017-06-16 15:50:54 -0700:
>> So I'm trying to figure out how to actually use it.
>>
>> We (and any other container based deploy..) will run into some
>> chicken/egg problem - you need to deploy container to generate big
>> yaml with defaults, then you need to overload it with your
>
> The config schema file (the "big YAML with defaults") should be part of
> the packaged software, so the deployment tool shouldn't need to generate
> it unless you're handling drivers that are not included in tree.

Right that's what I was missing, I guess we can generate these during
buildtime without big issues, then it will be embedded into container,
shouldn't be too hard change and would work for both source and
binary.
>> configurations, validate if they're not deprecated, run container with
>
> It doesn't do it today, but the thing that converts the input data to
> the INI file could automatically translate old option names to their new
> names.
>
>> this ansible role (or module...really doesn't matter), spit out final
>
> Why does the config file need to be generated inside a container?

Outside of container you don't have oslo or nova (python libs), so to
get access to these you need to do it inside container.

>> confg, lay it down, deploy container again. And that will have to be
>> done for every host class (as configs might differ host to host). Imho
>> a bit too much for this to be appealing (but I might be wrong). I'd
>> much rather have:
>> 1. Yaml as input to oslo.config instead of broken ini
>
> I'm not opposed to switching to YAML, but it's a bit more involved than
> just adding support in the parser. All of the work that has been done on
> generating sample default files and documentation needs to be updated to
> support YAML. We need a migration path to move everyone from INI to
> YAML. And we need to update devstack and all of its plugins to edit the
> new file format. There are probably more tasks involved in the
> migration. I'm dealing with a couple of other projects right now, and
> don't have time to plan all of that out myself. If someone else wants to
> pick it up, I can help with reviews on the spec and code changes.

Switching is a big no, everyone would hate us with emotion pure as
mountain spring water. It's to support both at same time, which makes
it slightly more complex. We could make full switch after few releases
of deprecation I guess. Anyway, agree, lots of work.

>
>> 2. Validator to throw an error if one of our regular,
>> template-rendered, configs is deprecated
>>
>> We can run this validator in gate to have quick feedback when
>> something gets deprecated.
>>
>> Thoughts?
>> Michal
>>
>> On 16 June 2017 at 13:24, Emilien Macchi <emilien at redhat.com> wrote:
>> > On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský <jistr at redhat.com> wrote:
>> >> On 15.6.2017 19:06, Emilien Macchi wrote:
>> >>>
>> >>> I missed [tripleo] tag.
>> >>>
>> >>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <emilien at redhat.com>
>> >>> wrote:
>> >>>>
>> >>>> If you haven't followed the "Configuration management with etcd /
>> >>>> confd" thread [1], Doug found out that using confd to generate
>> >>>> configuration files wouldn't work for the Cinder case where we don't
>> >>>> know in advance of the deployment what settings to tell confd to look
>> >>>> at.
>> >>>> We are still looking for a generic way to generate *.conf files for
>> >>>> OpenStack, that would be usable by Deployment tools and operators.
>> >>>> Right now, Doug and I are investigating some tooling that would be
>> >>>> useful to achieve this goal.
>> >>>>
>> >>>> Doug has prototyped an Ansible role that would generate configuration
>> >>>> files by consumming 2 things:
>> >>>>
>> >>>> * Configuration schema, generated by Ben's work with Machine Readable
>> >>>> Sample Config.
>> >>>>    $ oslo-config-generator --namespace cinder --format yaml >
>> >>>> cinder-schema.yaml
>> >>>>
>> >>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>> >>>> some extra data not included in the original version.
>> >>>>
>> >>>> * Parameters values provided in config_data directly in the playbook:
>> >>>>     config_data:
>> >>>>       DEFAULT:
>> >>>>         transport_url: rabbit://user:password@hostname
>> >>>>         verbose: true
>> >>>>
>> >>>> There are 2 options disabled by default but which would be useful for
>> >>>> production environments:
>> >>>> * Set to true to always show all configuration values:
>> >>>> config_show_defaults
>> >>>> * Set to true to show the help text: config_show_help: true
>> >>>>
>> >>>> The Ansible module is available on github:
>> >>>> https://github.com/dhellmann/oslo-config-ansible
>> >>>>
>> >>>> To try this out, just run:
>> >>>>    $ ansible-playbook ./playbook.yml
>> >>>>
>> >>>> You can quickly see the output of cinder.conf:
>> >>>>      https://clbin.com/HmS58
>> >>>>
>> >>>>
>> >>>> What are the next steps:
>> >>>>
>> >>>> * Getting feedback from Deployment Tools and operators on the concept
>> >>>> of this module.
>> >>>>    Maybe this module could replace what is done by Kolla with
>> >>>> merge_configs and OpenStack Ansible with config_template.
>> >>>> * On the TripleO side, we would like to see if this module could
>> >>>> replace the Puppet OpenStack modules that are now mostly used for
>> >>>> generating configuration files for containers.
>> >>>>    A transition path would be having Heat to generate Ansible vars
>> >>>> files and give it to this module. We could integrate the playbook into
>> >>>> a new task in the composable services, something like
>> >>>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>> >>>> also driven by Ansible.
>> >>
>> >>
>> >> This sounds good to me, though one issue i can presently see is that Puppet
>> >> modules sometimes contain quite a bit of data processing logic ("smart"
>> >> variables which map 1-to-N rather than 1-to-1 to actual config values, and
>> >> often not just in openstack service configs, e.g. puppet-nova also
>> >> configures libvirt, etc.). Also we use some non-config aspects from the
>> >> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
>> >> need to implement this functionality elsewhere when replacing the Puppet
>> >> modules. Not a blocker, but something to keep in mind.
>> >
>> > 2 interesting things:
>> >
>> > - For the logic that are done by puppet modules for some parameters:
>> > yes I agree, this problem isn't solved now. This thread talks about
>> > config management, with some data in entry, it's a very little step I
>> > know, but it's on purpose.
>> >   Once we figure how to do that, we can think about the data
>> > generation and where to put the logic (I think the logic is too
>> > opinionated to be in a common project, but I might be wrong).
>> >
>> > - Things like libvirt, mysql, etc will be managed by something else
>> > but Puppet I think; this is out of topic for now. For Keystone
>> > resources, same thing, we could use some native python clients or
>> > Ansible modules if we switch to Ansible, etc.
>> >
>> > Again, topic is really on "give me an ini file".
>> >
>> >>>> * Another similar option to what Doug did is to write a standalone
>> >>>> tool that would generate configuration, and for Ansible users we would
>> >>>> write a new module to use this tool.
>> >>>>    Example:
>> >>>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
>> >>>> cinder-schema.yaml (note this tool already exists)
>> >>>>        Step 2. Create config_data.yaml in a specific format with
>> >>>> parameters values for what we want to configure (note this format
>> >>>> doesn't exist yet but look at what Doug did in the role, we could use
>> >>>> the same kind of schema).
>> >>>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>> >>>> cinder.conf (note this tool doesn't exist yet)
>> >>
>> >>
>> >> +1 on standalone tool which can be used in different contexts (by different
>> >> higher level tools), this sounds generally useful.
>> >
>> > Ack, good feedback.
>> >
>> >>>>
>> >>>>        For Ansible users, we would write an Ansible module that would
>> >>>> take in entry 2 files: the schema and the data. The module would just
>> >>>> run the tool provided by oslo.config.
>> >>>>        Example:
>> >>>>            - name: Generate cinder.conf
>> >>>>              oslo-gen-config: schema=cinder-schema.yaml
>> >>>>                                         data=config_data.yaml
>> >>
>> >>
>> >> +1 for module rather than a role. "Take these inputs and produce that
>> >> output" fits the module semantics better than role semantics IMO.
>> >>
>> >> FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
>> >> containers could result in a nicer/safer/more-debuggable containerized
>> >> OpenStack setup than etcd + confd in daemon mode + mutable-config
>> >> containers.
>> >
>> > Good feedback, I'm also looking forward to hear from Kolla, OSA and Helm folks.
>> >
>> >>>>
>> >>>>
>> >>>> Please bring feedback and thoughts, it's really important to know what
>> >>>> folks from Installers think about this idea; again the ultimate goal
>> >>>> is to provide a reference tool to generate configuration in OpenStack,
>> >>>> in a way that scales and is friendly for our operators.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> [1]
>> >>>> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
>> >>>> --
>> >>>> Emilien Macchi
>> >>>
>> >>>
>> >>>
>> >>>
>> >>
>> >> Have a good day,
>> >>
>> >> Jirka
>> >>
>> >> __________________________________________________________________________
>> >> OpenStack Development Mailing List (not for usage questions)
>> >> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> > --
>> > Emilien Macchi
>> >
>> > __________________________________________________________________________
>> > OpenStack Development Mailing List (not for usage questions)
>> > Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev