Open Stack

So I'm trying to figure out how to actually use it.

We (and any other container based deploy..) will run into some
chicken/egg problem - you need to deploy container to generate big
yaml with defaults, then you need to overload it with your
configurations, validate if they're not deprecated, run container with
this ansible role (or module...really doesn't matter), spit out final
confg, lay it down, deploy container again. And that will have to be
done for every host class (as configs might differ host to host). Imho
a bit too much for this to be appealing (but I might be wrong). I'd
much rather have:
1. Yaml as input to oslo.config instead of broken ini
2. Validator to throw an error if one of our regular,
template-rendered, configs is deprecated

We can run this validator in gate to have quick feedback when
something gets deprecated.

Thoughts?
Michal

On 16 June 2017 at 13:24, Emilien Macchi <emilien at redhat.com> wrote:
> On Fri, Jun 16, 2017 at 11:09 AM, Jiří Stránský <jistr at redhat.com> wrote:
>> On 15.6.2017 19:06, Emilien Macchi wrote:
>>>
>>> I missed [tripleo] tag.
>>>
>>> On Thu, Jun 15, 2017 at 12:09 PM, Emilien Macchi <emilien at redhat.com>
>>> wrote:
>>>>
>>>> If you haven't followed the "Configuration management with etcd /
>>>> confd" thread [1], Doug found out that using confd to generate
>>>> configuration files wouldn't work for the Cinder case where we don't
>>>> know in advance of the deployment what settings to tell confd to look
>>>> at.
>>>> We are still looking for a generic way to generate *.conf files for
>>>> OpenStack, that would be usable by Deployment tools and operators.
>>>> Right now, Doug and I are investigating some tooling that would be
>>>> useful to achieve this goal.
>>>>
>>>> Doug has prototyped an Ansible role that would generate configuration
>>>> files by consumming 2 things:
>>>>
>>>> * Configuration schema, generated by Ben's work with Machine Readable
>>>> Sample Config.
>>>>    $ oslo-config-generator --namespace cinder --format yaml >
>>>> cinder-schema.yaml
>>>>
>>>> It also needs: https://review.openstack.org/#/c/474306/ to generate
>>>> some extra data not included in the original version.
>>>>
>>>> * Parameters values provided in config_data directly in the playbook:
>>>>     config_data:
>>>>       DEFAULT:
>>>>         transport_url: rabbit://user:password@hostname
>>>>         verbose: true
>>>>
>>>> There are 2 options disabled by default but which would be useful for
>>>> production environments:
>>>> * Set to true to always show all configuration values:
>>>> config_show_defaults
>>>> * Set to true to show the help text: config_show_help: true
>>>>
>>>> The Ansible module is available on github:
>>>> https://github.com/dhellmann/oslo-config-ansible
>>>>
>>>> To try this out, just run:
>>>>    $ ansible-playbook ./playbook.yml
>>>>
>>>> You can quickly see the output of cinder.conf:
>>>>      https://clbin.com/HmS58
>>>>
>>>>
>>>> What are the next steps:
>>>>
>>>> * Getting feedback from Deployment Tools and operators on the concept
>>>> of this module.
>>>>    Maybe this module could replace what is done by Kolla with
>>>> merge_configs and OpenStack Ansible with config_template.
>>>> * On the TripleO side, we would like to see if this module could
>>>> replace the Puppet OpenStack modules that are now mostly used for
>>>> generating configuration files for containers.
>>>>    A transition path would be having Heat to generate Ansible vars
>>>> files and give it to this module. We could integrate the playbook into
>>>> a new task in the composable services, something like
>>>>    "os_gen_config_tasks", a bit like we already have for upgrade tasks,
>>>> also driven by Ansible.
>>
>>
>> This sounds good to me, though one issue i can presently see is that Puppet
>> modules sometimes contain quite a bit of data processing logic ("smart"
>> variables which map 1-to-N rather than 1-to-1 to actual config values, and
>> often not just in openstack service configs, e.g. puppet-nova also
>> configures libvirt, etc.). Also we use some non-config aspects from the
>> Puppet modules (e.g. seeding Keystone tenants/services/endpoints/...). We'd
>> need to implement this functionality elsewhere when replacing the Puppet
>> modules. Not a blocker, but something to keep in mind.
>
> 2 interesting things:
>
> - For the logic that are done by puppet modules for some parameters:
> yes I agree, this problem isn't solved now. This thread talks about
> config management, with some data in entry, it's a very little step I
> know, but it's on purpose.
>   Once we figure how to do that, we can think about the data
> generation and where to put the logic (I think the logic is too
> opinionated to be in a common project, but I might be wrong).
>
> - Things like libvirt, mysql, etc will be managed by something else
> but Puppet I think; this is out of topic for now. For Keystone
> resources, same thing, we could use some native python clients or
> Ansible modules if we switch to Ansible, etc.
>
> Again, topic is really on "give me an ini file".
>
>>>> * Another similar option to what Doug did is to write a standalone
>>>> tool that would generate configuration, and for Ansible users we would
>>>> write a new module to use this tool.
>>>>    Example:
>>>>        Step 1. oslo-config-generator --namespace cinder --format yaml >
>>>> cinder-schema.yaml (note this tool already exists)
>>>>        Step 2. Create config_data.yaml in a specific format with
>>>> parameters values for what we want to configure (note this format
>>>> doesn't exist yet but look at what Doug did in the role, we could use
>>>> the same kind of schema).
>>>>        Step 3. oslo-gen-config -i config_data.yaml -s schema.yaml >
>>>> cinder.conf (note this tool doesn't exist yet)
>>
>>
>> +1 on standalone tool which can be used in different contexts (by different
>> higher level tools), this sounds generally useful.
>
> Ack, good feedback.
>
>>>>
>>>>        For Ansible users, we would write an Ansible module that would
>>>> take in entry 2 files: the schema and the data. The module would just
>>>> run the tool provided by oslo.config.
>>>>        Example:
>>>>            - name: Generate cinder.conf
>>>>              oslo-gen-config: schema=cinder-schema.yaml
>>>>                                         data=config_data.yaml
>>
>>
>> +1 for module rather than a role. "Take these inputs and produce that
>> output" fits the module semantics better than role semantics IMO.
>>
>> FWIW as i see it right now, this ^^ + ConfigMaps + immutable-config
>> containers could result in a nicer/safer/more-debuggable containerized
>> OpenStack setup than etcd + confd in daemon mode + mutable-config
>> containers.
>
> Good feedback, I'm also looking forward to hear from Kolla, OSA and Helm folks.
>
>>>>
>>>>
>>>> Please bring feedback and thoughts, it's really important to know what
>>>> folks from Installers think about this idea; again the ultimate goal
>>>> is to provide a reference tool to generate configuration in OpenStack,
>>>> in a way that scales and is friendly for our operators.
>>>>
>>>> Thanks,
>>>>
>>>> [1]
>>>> http://lists.openstack.org/pipermail/openstack-dev/2017-June/118176.html
>>>> --
>>>> Emilien Macchi
>>>
>>>
>>>
>>>
>>
>> Have a good day,
>>
>> Jirka
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> Emilien Macchi
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev