[openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

Steven Dake (stdake) stdake at cisco.com
Fri May 27 05:58:58 UTC 2016

On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net> wrote:

>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) <stdake at cisco.com>
>> Hey folks,
>> While Swapnil has been busy churning the dockerfile.j2 files to all
>> the same style, and we also had summit where we declared we would solve
>> plugin problem, I have decided to begin work on a DSL prototype.
>> Here are the problems I want to solve in order of importance by this
>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>> Provide a programmatic way to manage Dockerfile construction rather
>>then a
>> manual (with vi or emacs or the like) mechanism
>> Allow complete overrides of every facet of Dockerfile construction, most
>> especially repositories per container (rather than in the base
>>container) to
>> permit the use case of dependencies from one version with dependencies
>> another version of a different service
>> Get out of the business of maintaining 100+ dockerfiles but instead
>> one master file which defines the data that needs to be used to
>> Dockerfiles
>> Permit different types of optimizations or Dockerfile building by
>> around the parser implementation ­ to allow layering of each operation,
>> alternatively to merge layers as we do today
>> I don't believe we can proceed with both binary and source plugins
>>given our
>> current implementation of Dockerfiles in any sane way.
>> I further don't believe it is possible to customize repositories &
>> files per container, which I receive increasing requests for offline.
>> To that end, I've created a very very rough prototype which builds the
>> container as well as a mariadb container.  The mariadb container builds
>> I suspect would work.
>> An example of the DSL usage is here:
>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>> A very poorly written parser is here:
>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>> I played around with INI as a format, to take advantage of oslo.config
>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>> I'd appreciate reviews on the YML implementation especially.
>> How I see this work progressing is as follows:
>> A yml file describing all docker containers for all distros is placed in
>> kolla/docker
>> The build tool adds an option ‹use-yml which uses the YML file
>> A parser (such as load.py above) is integrated into build.py to lay
>>down he
>> Dockerfiles
>> Wait 4-6 weeks for people to find bugs and complain
>> Make the ‹use-yml the default for 4-6 weeks
>> Once we feel confident in the yml implementation, remove all
>> files
>> Remove ‹use-yml option
>> Remove all jinja2-isms from build.py
>> This is similar to the work that took place to convert from raw
>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>> will be the last major refactor of the dockerfiles unless someone has
>> significant complaints about the approach.
>> Regards
>> -steve
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>The DSL template to generate the Dockerfile seems way better than the
>jinja templates in terms of extension which is currently a major
>bottleneck in the plugin implementation. I am +2+W on this plan of
>action to test it for next 4-6 weeks and see thereon.


Customization and plugins are the trigger for the work.  I was thinking of
the following:

Elemental.yml (ships with Kolla)
Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
merged with elemental.yml)
Elemental-override.yml (operator provides in /etc/kolla, this file
overrides any YAML sections defined)

I think merging and overriding the yaml files should be pretty easy,
compared to jinja2, where I don't even know where to begin in a way that
the operator doesn't have to have deep knowledge of Kolla's internal

>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe

More information about the OpenStack-dev mailing list