[openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

Michał Jastrzębski inc007 at gmail.com
Tue May 31 20:42:22 UTC 2016

I am opposed to this idea as I don't think we need this. We can solve
many problems by using jinja2 to greater extend. I'll publish demo of
few improvements soon, please bear with me before we make a
arch-changing call.

On 29 May 2016 at 14:41, Steven Dake (stdake) <stdake at cisco.com> wrote:
>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:
>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net>
>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) <stdake at cisco.com>
>>>>> Hey folks,
>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>>> the same style, and we also had summit where we declared we would
>>>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>>> Here are the problems I want to solve in order of importance by this
>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>>>> Provide a programmatic way to manage Dockerfile construction rather
>>>>>then a
>>>>> manual (with vi or emacs or the like) mechanism
>>>>> Allow complete overrides of every facet of Dockerfile construction,
>>>>> especially repositories per container (rather than in the base
>>>>>container) to
>>>>> permit the use case of dependencies from one version with dependencies
>>>>> another version of a different service
>>>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>>> one master file which defines the data that needs to be used to
>>>>> Dockerfiles
>>>>> Permit different types of optimizations or Dockerfile building by
>>>>> around the parser implementation ­ to allow layering of each
>>>>> alternatively to merge layers as we do today
>>>>> I don't believe we can proceed with both binary and source plugins
>>>>>given our
>>>>> current implementation of Dockerfiles in any sane way.
>>>>> I further don't believe it is possible to customize repositories &
>>>>> files per container, which I receive increasing requests for offline.
>>>>> To that end, I've created a very very rough prototype which builds the
>>>>> container as well as a mariadb container.  The mariadb container
>>>>> I suspect would work.
>>>>> An example of the DSL usage is here:
>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>>> A very poorly written parser is here:
>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>>> I played around with INI as a format, to take advantage of oslo.config
>>>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>>> I'd appreciate reviews on the YML implementation especially.
>>>>> How I see this work progressing is as follows:
>>>>> A yml file describing all docker containers for all distros is placed
>>>>> kolla/docker
>>>>> The build tool adds an option ‹use-yml which uses the YML file
>>>>> A parser (such as load.py above) is integrated into build.py to lay
>>>>>down he
>>>>> Dockerfiles
>>>>> Wait 4-6 weeks for people to find bugs and complain
>>>>> Make the ‹use-yml the default for 4-6 weeks
>>>>> Once we feel confident in the yml implementation, remove all
>>>>> files
>>>>> Remove ‹use-yml option
>>>>> Remove all jinja2-isms from build.py
>>>>> This is similar to the work that took place to convert from raw
>>>>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>>>>> will be the last major refactor of the dockerfiles unless someone has
>>>>> significant complaints about the approach.
>>>>> Regards
>>>>> -steve
> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhouser at cisco.com> wrote:
>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>>be, so most of what you're saying is going over my head.  I think mainly
>>I don't understand the problem statement.  It looks like you're pulling
>>all the "hard coded" things out of the docker files, and making them user
>>replaceable?  So the dockerfiles just become a list of required steps,
>>and the user can change how each step is implemented?  Would this also
>>unify the dockefiles so there wouldn't be a huge if statements between
>>Centos and Ubuntu?
> What is being pulled out is all of the metadata used by the Dockerfiles or
> Kolla in general.  This metadata, being structured either as a dictionary
> or ordered list, can be manipulated by simple python tools to do things
> like merge sections and override sections or optimize the built images.
> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
> image produced by the parser.  The jinja2 templates we have today cannot
> be easily overridden.  We have to provide a new key for each type of
> override, which is super onerous on the build.py tool.
> To your question of docker files being a list of required steps, with this
> method Dockerfile.j2 would go away permanently.  On each bulid, as is done
> now to process a jinja2 file into a Dockerfile, the elemental.yml file
> would be processed into a Dockerfile for that particular set of build
> options.
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>The DSL template to generate the Dockerfile seems way better than the
>>>>jinja templates in terms of extension which is currently a major
>>>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>>>action to test it for next 4-6 weeks and see thereon.
>>>Customization and plugins are the trigger for the work.  I was thinking
>>>the following:
>>>Elemental.yml (ships with Kolla)
>>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>>>merged with elemental.yml)
>>>Elemental-override.yml (operator provides in /etc/kolla, this file
>>>overrides any YAML sections defined)
>>>I think merging and overriding the yaml files should be pretty easy,
>>>compared to jinja2, where I don't even know where to begin in a way that
>>>the operator doesn't have to have deep knowledge of Kolla's internal
>>>>OpenStack Development Mailing List (not for usage questions)
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>OpenStack Development Mailing List (not for usage questions)
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

More information about the OpenStack-dev mailing list