[openstack-dev] [kolla] prototype of a DSL for generating Dockerfiles

Steven Dake (stdake) stdake at cisco.com
Wed Jun 1 00:36:05 UTC 2016



On 5/31/16, 1:42 PM, "Michał Jastrzębski" <inc007 at gmail.com> wrote:

>I am opposed to this idea as I don't think we need this. We can solve
>many problems by using jinja2 to greater extend. I'll publish demo of
>few improvements soon, please bear with me before we make a
>arch-changing call.

Can you make a specification please as you have asked me to do?

>
>On 29 May 2016 at 14:41, Steven Dake (stdake) <stdake at cisco.com> wrote:
>>
>>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:
>>>
>>>>
>>>>
>>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net>
>>>>wrote:
>>>>
>>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)
>>>>><stdake at cisco.com>
>>>>>wrote:
>>>>>> Hey folks,
>>>>>>
>>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>>>>match
>>>>>> the same style, and we also had summit where we declared we would
>>>>>>solve
>>>>>>the
>>>>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>>>>
>>>>>> Here are the problems I want to solve in order of importance by this
>>>>>>work:
>>>>>>
>>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>>>>> Provide a programmatic way to manage Dockerfile construction rather
>>>>>>then a
>>>>>> manual (with vi or emacs or the like) mechanism
>>>>>> Allow complete overrides of every facet of Dockerfile construction,
>>>>>>most
>>>>>> especially repositories per container (rather than in the base
>>>>>>container) to
>>>>>> permit the use case of dependencies from one version with
>>>>>>dependencies
>>>>>>in
>>>>>> another version of a different service
>>>>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>>>>maintain
>>>>>> one master file which defines the data that needs to be used to
>>>>>>construct
>>>>>> Dockerfiles
>>>>>> Permit different types of optimizations or Dockerfile building by
>>>>>>changing
>>>>>> around the parser implementation ­ to allow layering of each
>>>>>>operation,
>>>>>>or
>>>>>> alternatively to merge layers as we do today
>>>>>>
>>>>>> I don't believe we can proceed with both binary and source plugins
>>>>>>given our
>>>>>> current implementation of Dockerfiles in any sane way.
>>>>>>
>>>>>> I further don't believe it is possible to customize repositories &
>>>>>>installed
>>>>>> files per container, which I receive increasing requests for
>>>>>>offline.
>>>>>>
>>>>>> To that end, I've created a very very rough prototype which builds
>>>>>>the
>>>>>>base
>>>>>> container as well as a mariadb container.  The mariadb container
>>>>>>builds
>>>>>>and
>>>>>> I suspect would work.
>>>>>>
>>>>>> An example of the DSL usage is here:
>>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>>>>
>>>>>> A very poorly written parser is here:
>>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>>>>
>>>>>> I played around with INI as a format, to take advantage of
>>>>>>oslo.config
>>>>>>and
>>>>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>>>>
>>>>>> I'd appreciate reviews on the YML implementation especially.
>>>>>>
>>>>>> How I see this work progressing is as follows:
>>>>>>
>>>>>> A yml file describing all docker containers for all distros is
>>>>>>placed
>>>>>>in
>>>>>> kolla/docker
>>>>>> The build tool adds an option ‹use-yml which uses the YML file
>>>>>> A parser (such as load.py above) is integrated into build.py to lay
>>>>>>down he
>>>>>> Dockerfiles
>>>>>> Wait 4-6 weeks for people to find bugs and complain
>>>>>> Make the ‹use-yml the default for 4-6 weeks
>>>>>> Once we feel confident in the yml implementation, remove all
>>>>>>Dockerfile.j2
>>>>>> files
>>>>>> Remove ‹use-yml option
>>>>>> Remove all jinja2-isms from build.py
>>>>>>
>>>>>> This is similar to the work that took place to convert from raw
>>>>>>Dockerfiles
>>>>>> to Dockerfile.j2 files.  We are just reusing that pattern.
>>>>>>Hopefully
>>>>>>this
>>>>>> will be the last major refactor of the dockerfiles unless someone
>>>>>>has
>>>>>>some
>>>>>> significant complaints about the approach.
>>>>>>
>>>>>> Regards
>>>>>> -steve

Hey folks,

I have produced a specification for Kolla's DSL (which I call Elemental).
The spec is ready for review here:
https://review.openstack.org/#/c/323612/


Regards
-steve

>>
>>
>> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhouser at cisco.com> wrote:
>>
>>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>>>be, so most of what you're saying is going over my head.  I think mainly
>>>I don't understand the problem statement.  It looks like you're pulling
>>>all the "hard coded" things out of the docker files, and making them
>>>user
>>>replaceable?  So the dockerfiles just become a list of required steps,
>>>and the user can change how each step is implemented?  Would this also
>>>unify the dockefiles so there wouldn't be a huge if statements between
>>>Centos and Ubuntu?
>>>
>>>Thx,
>>>Britt
>>>
>>
>> What is being pulled out is all of the metadata used by the Dockerfiles
>>or
>> Kolla in general.  This metadata, being structured either as a
>>dictionary
>> or ordered list, can be manipulated by simple python tools to do things
>> like merge sections and override sections or optimize the built images.
>> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
>> image produced by the parser.  The jinja2 templates we have today cannot
>> be easily overridden.  We have to provide a new key for each type of
>> override, which is super onerous on the build.py tool.
>>
>> To your question of docker files being a list of required steps, with
>>this
>> method Dockerfile.j2 would go away permanently.  On each bulid, as is
>>done
>> now to process a jinja2 file into a Dockerfile, the elemental.yml file
>> would be processed into a Dockerfile for that particular set of build
>> options.
>>>
>>
>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>_____________________________________________________________________
>>>>>>__
>>>>>>__
>>>>>>_
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>>The DSL template to generate the Dockerfile seems way better than the
>>>>>jinja templates in terms of extension which is currently a major
>>>>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>>>>action to test it for next 4-6 weeks and see thereon.
>>>>>
>>>>>Swapnil
>>>>>
>>>>
>>>>Agree.
>>>>
>>>>Customization and plugins are the trigger for the work.  I was thinking
>>>>of
>>>>the following:
>>>>
>>>>Elemental.yml (ships with Kolla)
>>>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>>>>merged with elemental.yml)
>>>>Elemental-override.yml (operator provides in /etc/kolla, this file
>>>>overrides any YAML sections defined)
>>>>
>>>>I think merging and overriding the yaml files should be pretty easy,
>>>>compared to jinja2, where I don't even know where to begin in a way
>>>>that
>>>>the operator doesn't have to have deep knowledge of Kolla's internal
>>>>implementation.
>>>>
>>>>Regards
>>>>-steve
>>>>
>>>>>______________________________________________________________________
>>>>>__
>>>>>__
>>>>>OpenStack Development Mailing List (not for usage questions)
>>>>>Unsubscribe:
>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>_______________________________________________________________________
>>>>__
>>>>_
>>>>OpenStack Development Mailing List (not for usage questions)
>>>>Unsubscribe:
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>________________________________________________________________________
>>>__
>>>OpenStack Development Mailing List (not for usage questions)
>>>Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> 
>>_________________________________________________________________________
>>_
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list