[openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

Britt Houser (bhouser) bhouser at cisco.com
Fri May 27 10:44:17 UTC 2016

I admit I'm not as knowledgable about the Kolla codebase as I'd like to be, so most of what you're saying is going over my head.  I think mainly I don't understand the problem statement.  It looks like you're pulling all the "hard coded" things out of the docker files, and making them user replaceable?  So the dockerfiles just become a list of required steps, and the user can change how each step is implemented?  Would this also unify the dockefiles so there wouldn't be a huge if statements between Centos and Ubuntu?


On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:

>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net> wrote:
>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) <stdake at cisco.com>
>>> Hey folks,
>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>> the same style, and we also had summit where we declared we would solve
>>> plugin problem, I have decided to begin work on a DSL prototype.
>>> Here are the problems I want to solve in order of importance by this
>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>> Provide a programmatic way to manage Dockerfile construction rather
>>>then a
>>> manual (with vi or emacs or the like) mechanism
>>> Allow complete overrides of every facet of Dockerfile construction, most
>>> especially repositories per container (rather than in the base
>>>container) to
>>> permit the use case of dependencies from one version with dependencies
>>> another version of a different service
>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>> one master file which defines the data that needs to be used to
>>> Dockerfiles
>>> Permit different types of optimizations or Dockerfile building by
>>> around the parser implementation ­ to allow layering of each operation,
>>> alternatively to merge layers as we do today
>>> I don't believe we can proceed with both binary and source plugins
>>>given our
>>> current implementation of Dockerfiles in any sane way.
>>> I further don't believe it is possible to customize repositories &
>>> files per container, which I receive increasing requests for offline.
>>> To that end, I've created a very very rough prototype which builds the
>>> container as well as a mariadb container.  The mariadb container builds
>>> I suspect would work.
>>> An example of the DSL usage is here:
>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>> A very poorly written parser is here:
>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>> I played around with INI as a format, to take advantage of oslo.config
>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>> I'd appreciate reviews on the YML implementation especially.
>>> How I see this work progressing is as follows:
>>> A yml file describing all docker containers for all distros is placed in
>>> kolla/docker
>>> The build tool adds an option ‹use-yml which uses the YML file
>>> A parser (such as load.py above) is integrated into build.py to lay
>>>down he
>>> Dockerfiles
>>> Wait 4-6 weeks for people to find bugs and complain
>>> Make the ‹use-yml the default for 4-6 weeks
>>> Once we feel confident in the yml implementation, remove all
>>> files
>>> Remove ‹use-yml option
>>> Remove all jinja2-isms from build.py
>>> This is similar to the work that took place to convert from raw
>>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>>> will be the last major refactor of the dockerfiles unless someone has
>>> significant complaints about the approach.
>>> Regards
>>> -steve
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>The DSL template to generate the Dockerfile seems way better than the
>>jinja templates in terms of extension which is currently a major
>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>action to test it for next 4-6 weeks and see thereon.
>Customization and plugins are the trigger for the work.  I was thinking of
>the following:
>Elemental.yml (ships with Kolla)
>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>merged with elemental.yml)
>Elemental-override.yml (operator provides in /etc/kolla, this file
>overrides any YAML sections defined)
>I think merging and overriding the yaml files should be pretty easy,
>compared to jinja2, where I don't even know where to begin in a way that
>the operator doesn't have to have deep knowledge of Kolla's internal
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe

More information about the OpenStack-dev mailing list