Open Stack

So this is prototype of working template overrides:
https://review.openstack.org/#/c/323612/

Pass --template-overrides=path-to-file to build.py
in file override you can add any custom code/logic/dockerfile stuff to
any of hooks we provide in Dockerfiles, and we'll provide a lot of
them as it's free and non breaking operation. With enough block you'll
be able to do virtually anything with any of the containers.

This one is already working. Only work needed is to provide more
hooks/continue with refactoring of dockerfiles.

Cheers,
Michal

On 31 May 2016 at 19:36, Steven Dake (stdake) <stdake at cisco.com> wrote:
>
>
> On 5/31/16, 1:42 PM, "Michał Jastrzębski" <inc007 at gmail.com> wrote:
>
>>I am opposed to this idea as I don't think we need this. We can solve
>>many problems by using jinja2 to greater extend. I'll publish demo of
>>few improvements soon, please bear with me before we make a
>>arch-changing call.
>
> Can you make a specification please as you have asked me to do?
>
>>
>>On 29 May 2016 at 14:41, Steven Dake (stdake) <stdake at cisco.com> wrote:
>>>
>>>>On 5/27/16, 1:58 AM, "Steven Dake (stdake)" <stdake at cisco.com> wrote:
>>>>
>>>>>
>>>>>
>>>>>On 5/26/16, 8:45 PM, "Swapnil Kulkarni (coolsvap)" <me at coolsvap.net>
>>>>>wrote:
>>>>>
>>>>>>On Fri, May 27, 2016 at 8:35 AM, Steven Dake (stdake)
>>>>>><stdake at cisco.com>
>>>>>>wrote:
>>>>>>> Hey folks,
>>>>>>>
>>>>>>> While Swapnil has been busy churning the dockerfile.j2 files to all
>>>>>>>match
>>>>>>> the same style, and we also had summit where we declared we would
>>>>>>>solve
>>>>>>>the
>>>>>>> plugin problem, I have decided to begin work on a DSL prototype.
>>>>>>>
>>>>>>> Here are the problems I want to solve in order of importance by this
>>>>>>>work:
>>>>>>>
>>>>>>> Build CentOS, Ubuntu, Oracle Linux, Debian, Fedora containers
>>>>>>> Provide a programmatic way to manage Dockerfile construction rather
>>>>>>>then a
>>>>>>> manual (with vi or emacs or the like) mechanism
>>>>>>> Allow complete overrides of every facet of Dockerfile construction,
>>>>>>>most
>>>>>>> especially repositories per container (rather than in the base
>>>>>>>container) to
>>>>>>> permit the use case of dependencies from one version with
>>>>>>>dependencies
>>>>>>>in
>>>>>>> another version of a different service
>>>>>>> Get out of the business of maintaining 100+ dockerfiles but instead
>>>>>>>maintain
>>>>>>> one master file which defines the data that needs to be used to
>>>>>>>construct
>>>>>>> Dockerfiles
>>>>>>> Permit different types of optimizations or Dockerfile building by
>>>>>>>changing
>>>>>>> around the parser implementation ­ to allow layering of each
>>>>>>>operation,
>>>>>>>or
>>>>>>> alternatively to merge layers as we do today
>>>>>>>
>>>>>>> I don't believe we can proceed with both binary and source plugins
>>>>>>>given our
>>>>>>> current implementation of Dockerfiles in any sane way.
>>>>>>>
>>>>>>> I further don't believe it is possible to customize repositories &
>>>>>>>installed
>>>>>>> files per container, which I receive increasing requests for
>>>>>>>offline.
>>>>>>>
>>>>>>> To that end, I've created a very very rough prototype which builds
>>>>>>>the
>>>>>>>base
>>>>>>> container as well as a mariadb container.  The mariadb container
>>>>>>>builds
>>>>>>>and
>>>>>>> I suspect would work.
>>>>>>>
>>>>>>> An example of the DSL usage is here:
>>>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/dsl.yml
>>>>>>>
>>>>>>> A very poorly written parser is here:
>>>>>>> https://review.openstack.org/#/c/321468/4/dockerdsl/load.py
>>>>>>>
>>>>>>> I played around with INI as a format, to take advantage of
>>>>>>>oslo.config
>>>>>>>and
>>>>>>> kolla-build.conf, but that didn't work out.  YML is the way to go.
>>>>>>>
>>>>>>> I'd appreciate reviews on the YML implementation especially.
>>>>>>>
>>>>>>> How I see this work progressing is as follows:
>>>>>>>
>>>>>>> A yml file describing all docker containers for all distros is
>>>>>>>placed
>>>>>>>in
>>>>>>> kolla/docker
>>>>>>> The build tool adds an option ‹use-yml which uses the YML file
>>>>>>> A parser (such as load.py above) is integrated into build.py to lay
>>>>>>>down he
>>>>>>> Dockerfiles
>>>>>>> Wait 4-6 weeks for people to find bugs and complain
>>>>>>> Make the ‹use-yml the default for 4-6 weeks
>>>>>>> Once we feel confident in the yml implementation, remove all
>>>>>>>Dockerfile.j2
>>>>>>> files
>>>>>>> Remove ‹use-yml option
>>>>>>> Remove all jinja2-isms from build.py
>>>>>>>
>>>>>>> This is similar to the work that took place to convert from raw
>>>>>>>Dockerfiles
>>>>>>> to Dockerfile.j2 files.  We are just reusing that pattern.
>>>>>>>Hopefully
>>>>>>>this
>>>>>>> will be the last major refactor of the dockerfiles unless someone
>>>>>>>has
>>>>>>>some
>>>>>>> significant complaints about the approach.
>>>>>>>
>>>>>>> Regards
>>>>>>> -steve
>
> Hey folks,
>
> I have produced a specification for Kolla's DSL (which I call Elemental).
> The spec is ready for review here:
> https://review.openstack.org/#/c/323612/
>
>
> Regards
> -steve
>
>>>
>>>
>>> On 5/27/16, 3:44 AM, "Britt Houser (bhouser)" <bhouser at cisco.com> wrote:
>>>
>>>>I admit I'm not as knowledgable about the Kolla codebase as I'd like to
>>>>be, so most of what you're saying is going over my head.  I think mainly
>>>>I don't understand the problem statement.  It looks like you're pulling
>>>>all the "hard coded" things out of the docker files, and making them
>>>>user
>>>>replaceable?  So the dockerfiles just become a list of required steps,
>>>>and the user can change how each step is implemented?  Would this also
>>>>unify the dockefiles so there wouldn't be a huge if statements between
>>>>Centos and Ubuntu?
>>>>
>>>>Thx,
>>>>Britt
>>>>
>>>
>>> What is being pulled out is all of the metadata used by the Dockerfiles
>>>or
>>> Kolla in general.  This metadata, being structured either as a
>>>dictionary
>>> or ordered list, can be manipulated by simple python tools to do things
>>> like merge sections and override sections or optimize the built images.
>>> FWIW it looks without even trying the Dockerfiles produce a 50MB smaller
>>> image produced by the parser.  The jinja2 templates we have today cannot
>>> be easily overridden.  We have to provide a new key for each type of
>>> override, which is super onerous on the build.py tool.
>>>
>>> To your question of docker files being a list of required steps, with
>>>this
>>> method Dockerfile.j2 would go away permanently.  On each bulid, as is
>>>done
>>> now to process a jinja2 file into a Dockerfile, the elemental.yml file
>>> would be processed into a Dockerfile for that particular set of build
>>> options.
>>>>
>>>
>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>_____________________________________________________________________
>>>>>>>__
>>>>>>>__
>>>>>>>_
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>
>>>>>>The DSL template to generate the Dockerfile seems way better than the
>>>>>>jinja templates in terms of extension which is currently a major
>>>>>>bottleneck in the plugin implementation. I am +2+W on this plan of
>>>>>>action to test it for next 4-6 weeks and see thereon.
>>>>>>
>>>>>>Swapnil
>>>>>>
>>>>>
>>>>>Agree.
>>>>>
>>>>>Customization and plugins are the trigger for the work.  I was thinking
>>>>>of
>>>>>the following:
>>>>>
>>>>>Elemental.yml (ships with Kolla)
>>>>>Elemental-merge.yml (operator provides in /etc/kolla, this file is yaml
>>>>>merged with elemental.yml)
>>>>>Elemental-override.yml (operator provides in /etc/kolla, this file
>>>>>overrides any YAML sections defined)
>>>>>
>>>>>I think merging and overriding the yaml files should be pretty easy,
>>>>>compared to jinja2, where I don't even know where to begin in a way
>>>>>that
>>>>>the operator doesn't have to have deep knowledge of Kolla's internal
>>>>>implementation.
>>>>>
>>>>>Regards
>>>>>-steve
>>>>>
>>>>>>______________________________________________________________________
>>>>>>__
>>>>>>__
>>>>>>OpenStack Development Mailing List (not for usage questions)
>>>>>>Unsubscribe:
>>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>_______________________________________________________________________
>>>>>__
>>>>>_
>>>>>OpenStack Development Mailing List (not for usage questions)
>>>>>Unsubscribe:
>>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>________________________________________________________________________
>>>>__
>>>>OpenStack Development Mailing List (not for usage questions)
>>>>Unsubscribe:
>>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>_________________________________________________________________________
>>>_
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>>OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>__________________________________________________________________________
>>OpenStack Development Mailing List (not for usage questions)
>>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev