Open Stack

>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:
>
>>
>>
>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net>
>>wrote:
>>
>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) <stdake at cisco.com>
>>>wrote:
>>>> Hey folks,
>>>>
>>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>>match
>>>> the same style, and we also had summit where we declared we would
>>>>solve
>>>>the
>>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>>
>>>> Here are the problems I want to solve in order of importance by this
>>>>work:
>>>>
>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>>> Provide a programmatic way to manage Dockerfile construction rather
>>>>then a
>>>> manual (with vi or emacs or the like) mechanism
>>>> Allow complete overrides of every facet of Dockerfile construction,
>>>>most
>>>> especially repositories per container (rather than in the base
>>>>container) to
>>>> permit the use case of dependencies from one version with dependencies
>>>>in
>>>> another version of a different service
>>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>>maintain
>>>> one master file which defines the data that needs to be used to
>>>>construct
>>>> Dockerfiles
>>>> Permit different types of optimizations or Dockerfile building by
>>>>changing
>>>> around the parser implementation ­ to allow layering of each
>>>>operation,
>>>>or
>>>> alternatively to merge layers as we do today
>>>>
>>>> I don't believe we can proceed with both binary and source plugins
>>>>given our
>>>> current implementation of Dockerfiles in any sane way.
>>>>
>>>> I further don't believe it is possible to customize repositories &
>>>>installed
>>>> files per container, which I receive increasing requests for offline.
>>>>
>>>> To that end, I've created a very very rough prototype which builds the
>>>>base
>>>> container as well as a mariadb container.  The mariadb container
>>>>builds
>>>>and
>>>> I suspect would work.
>>>>
>>>> An example of the DSL usage is here:
>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>>
>>>> A very poorly written parser is here:
>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>>
>>>> I played around with INI as a format, to take advantage of oslo.config
>>>>and
>>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>>
>>>> I'd appreciate reviews on the YML implementation especially.
>>>>
>>>> How I see this work progressing is as follows:
>>>>
>>>> A yml file describing all docker containers for all distros is placed
>>>>in
>>>> kolla/docker
>>>> The build tool adds an option ‹use-yml which uses the YML file
>>>> A parser (such as load.py above) is integrated into build.py to lay
>>>>down he
>>>> Dockerfiles
>>>> Wait 4-6 weeks for people to find bugs and complain
>>>> Make the ‹use-yml the default for 4-6 weeks
>>>> Once we feel confident in the yml implementation, remove all
>>>>Dockerfile.j2
>>>> files
>>>> Remove ‹use-yml option
>>>> Remove all jinja2-isms from build.py
>>>>
>>>> This is similar to the work that took place to convert from raw
>>>>Dockerfiles
>>>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>>>>this
>>>> will be the last major refactor of the dockerfiles unless someone has
>>>>some
>>>> significant complaints about the approach.
>>>>
>>>> Regards
>>>> -steve


On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhouser at cisco.com> wrote:

>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>be, so most of what you're saying is going over my head.  I think mainly
>I don't understand the problem statement.  It looks like you're pulling
>all the "hard coded" things out of the docker files, and making them user
>replaceable?  So the dockerfiles just become a list of required steps,
>and the user can change how each step is implemented?  Would this also
>unify the dockefiles so there wouldn't be a huge if statements between
>Centos and Ubuntu?
>
>Thx,
>Britt
>

What is being pulled out is all of the metadata used by the Dockerfiles or
Kolla in general.  This metadata, being structured either as a dictionary
or ordered list, can be manipulated by simple python tools to do things
like merge sections and override sections or optimize the built images.
FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
image produced by the parser.  The jinja2 templates we have today cannot
be easily overridden.  We have to provide a new key for each type of
override, which is super onerous on the build.py tool.

To your question of docker files being a list of required steps, with this
method Dockerfile.j2 would go away permanently.  On each bulid, as is done
now to process a jinja2 file into a Dockerfile, the elemental.yml file
would be processed into a Dockerfile for that particular set of build
options. 
>


>>>>
>>>>
>>>> 
>>>>_______________________________________________________________________
>>>>__
>>>>_
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe: 
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>The DSL template to generate the Dockerfile seems way better than the
>>>jinja templates in terms of extension which is currently a major
>>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>>action to test it for next 4-6 weeks and see thereon.
>>>
>>>Swapnil
>>>
>>
>>Agree.
>>
>>Customization and plugins are the trigger for the work.  I was thinking
>>of
>>the following:
>>
>>Elemental.yml (ships with Kolla)
>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>>merged with elemental.yml)
>>Elemental-override.yml (operator provides in /etc/kolla, this file
>>overrides any YAML sections defined)
>>
>>I think merging and overriding the yaml files should be pretty easy,
>>compared to jinja2, where I don't even know where to begin in a way that
>>the operator doesn't have to have deep knowledge of Kolla's internal
>>implementation.
>>
>>Regards
>>-steve
>>  
>>>________________________________________________________________________
>>>__
>>>OpenStack Development Mailing List (not for usage questions)
>>>Unsubscribe: 
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>_________________________________________________________________________
>>_
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: 
>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev