Open Stack

I am opposed to this idea as I don't think we need this. We can solve
many problems by using jinja2 to greater extend. I'll publish demo of
few improvements soon, please bear with me before we make a
arch-changing call.

On 29 May 2016 at 14:41, Steven Dake (stdake) <stdake at cisco.com> wrote:
>
>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:
>>
>>>
>>>
>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net>
>>>wrote:
>>>
>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake) <stdake at cisco.com>
>>>>wrote:
>>>>> Hey folks,
>>>>>
>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>>>match
>>>>> the same style, and we also had summit where we declared we would
>>>>>solve
>>>>>the
>>>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>>>
>>>>> Here are the problems I want to solve in order of importance by this
>>>>>work:
>>>>>
>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>>>> Provide a programmatic way to manage Dockerfile construction rather
>>>>>then a
>>>>> manual (with vi or emacs or the like) mechanism
>>>>> Allow complete overrides of every facet of Dockerfile construction,
>>>>>most
>>>>> especially repositories per container (rather than in the base
>>>>>container) to
>>>>> permit the use case of dependencies from one version with dependencies
>>>>>in
>>>>> another version of a different service
>>>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>>>maintain
>>>>> one master file which defines the data that needs to be used to
>>>>>construct
>>>>> Dockerfiles
>>>>> Permit different types of optimizations or Dockerfile building by
>>>>>changing
>>>>> around the parser implementation ­ to allow layering of each
>>>>>operation,
>>>>>or
>>>>> alternatively to merge layers as we do today
>>>>>
>>>>> I don't believe we can proceed with both binary and source plugins
>>>>>given our
>>>>> current implementation of Dockerfiles in any sane way.
>>>>>
>>>>> I further don't believe it is possible to customize repositories &
>>>>>installed
>>>>> files per container, which I receive increasing requests for offline.
>>>>>
>>>>> To that end, I've created a very very rough prototype which builds the
>>>>>base
>>>>> container as well as a mariadb container.  The mariadb container
>>>>>builds
>>>>>and
>>>>> I suspect would work.
>>>>>
>>>>> An example of the DSL usage is here:
>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>>>
>>>>> A very poorly written parser is here:
>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>>>
>>>>> I played around with INI as a format, to take advantage of oslo.config
>>>>>and
>>>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>>>
>>>>> I'd appreciate reviews on the YML implementation especially.
>>>>>
>>>>> How I see this work progressing is as follows:
>>>>>
>>>>> A yml file describing all docker containers for all distros is placed
>>>>>in
>>>>> kolla/docker
>>>>> The build tool adds an option ‹use-yml which uses the YML file
>>>>> A parser (such as load.py above) is integrated into build.py to lay
>>>>>down he
>>>>> Dockerfiles
>>>>> Wait 4-6 weeks for people to find bugs and complain
>>>>> Make the ‹use-yml the default for 4-6 weeks
>>>>> Once we feel confident in the yml implementation, remove all
>>>>>Dockerfile.j2
>>>>> files
>>>>> Remove ‹use-yml option
>>>>> Remove all jinja2-isms from build.py
>>>>>
>>>>> This is similar to the work that took place to convert from raw
>>>>>Dockerfiles
>>>>> to Dockerfile.j2 files.  We are just reusing that pattern.  Hopefully
>>>>>this
>>>>> will be the last major refactor of the dockerfiles unless someone has
>>>>>some
>>>>> significant complaints about the approach.
>>>>>
>>>>> Regards
>>>>> -steve
>
>
> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhouser at cisco.com> wrote:
>
>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>>be, so most of what you're saying is going over my head.  I think mainly
>>I don't understand the problem statement.  It looks like you're pulling
>>all the "hard coded" things out of the docker files, and making them user
>>replaceable?  So the dockerfiles just become a list of required steps,
>>and the user can change how each step is implemented?  Would this also
>>unify the dockefiles so there wouldn't be a huge if statements between
>>Centos and Ubuntu?
>>
>>Thx,
>>Britt
>>
>
> What is being pulled out is all of the metadata used by the Dockerfiles or
> Kolla in general.  This metadata, being structured either as a dictionary
> or ordered list, can be manipulated by simple python tools to do things
> like merge sections and override sections or optimize the built images.
> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
> image produced by the parser.  The jinja2 templates we have today cannot
> be easily overridden.  We have to provide a new key for each type of
> override, which is super onerous on the build.py tool.
>
> To your question of docker files being a list of required steps, with this
> method Dockerfile.j2 would go away permanently.  On each bulid, as is done
> now to process a jinja2 file into a Dockerfile, the elemental.yml file
> would be processed into a Dockerfile for that particular set of build
> options.
>>
>
>
>>>>>
>>>>>
>>>>>
>>>>>_______________________________________________________________________
>>>>>__
>>>>>_
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>The DSL template to generate the Dockerfile seems way better than the
>>>>jinja templates in terms of extension which is currently a major
>>>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>>>action to test it for next 4-6 weeks and see thereon.
>>>>
>>>>Swapnil
>>>>
>>>
>>>Agree.
>>>
>>>Customization and plugins are the trigger for the work.  I was thinking
>>>of
>>>the following:
>>>
>>>Elemental.yml (ships with Kolla)
>>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>>>merged with elemental.yml)
>>>Elemental-override.yml (operator provides in /etc/kolla, this file
>>>overrides any YAML sections defined)
>>>
>>>I think merging and overriding the yaml files should be pretty easy,
>>>compared to jinja2, where I don't even know where to begin in a way that
>>>the operator doesn't have to have deep knowledge of Kolla's internal
>>>implementation.
>>>
>>>Regards
>>>-steve
>>>
>>>>________________________________________________________________________
>>>>__
>>>>OpenStack Development Mailing List (not for usage questions)
>>>>Unsubscribe:
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>_________________________________________________________________________
>>>_
>>>OpenStack Development Mailing List (not for usage questions)
>>>Unsubscribe:
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>__________________________________________________________________________
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev