On Wed, 2020-04-22 at 15:24 +0200, Radosław Piliszek wrote:
On Tue, Apr 21, 2020 at 9:12 PM Mohammed Naser <mnaser@vexxhost.com> wrote:
On Tue, Apr 21, 2020 at 5:15 AM Mark Goddard <mark@stackhpc.com> wrote:
On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
The above effort could well be coordinated with different projects to reuse bindep contents. So far Kolla does not use bindep because it often installs too much and not enough at the same time. Do note it would still be bindep-less for external services.
I like the idea of using bindep for source images, in combination with multi-stage builds. If we could reduce the length of our package dependency lists by relying on bindep, that would be a nice win. Ideally, between distro package dependencies and bindep, we would need to specify few additional packages.
I think this is a golden ticket of what I'm trying to accomplish and this is what OpenDev is currently doing. Ideally, there _should_ be no distro package dependencies because they should all be modeled inside bindep (using different profiles if needed).
I still think that if we do this, there is a lot of value in being able to do this in-repo because those Dockerfile's will be pretty static. While I appreciate that Kolla has all sorts of neat things to customize images, for a certain operator base, they'd much rather either have a Dockerfile to build off of or an existing base image somewhere.
Kolla could be the thing that drives this, Kolla could become "OpenStack containers", or parts of it could be. However, I think the "customization bits" should ideally stay outside of the scope of this, because we just want to publish an artifact with dependencies installed.
I think we all agree here that bindep is the way to go and whatever we decide on should include "better bindep experience". Could we collect together some concrete ideas how to make the bindep better? (and then suddenly bindep sig appears out of nowhere :O ) I guess Kolla is a great source especially for optional runtime bindep entries, though it can really only offer names of packages per distro rather than insight why they are used (except for git archaelogy of course). We could collaborate with each project to discover real relations between these. Making bindep better would benefit all efforts of deployment (with devstack included) so it's a win-win.
For the purpose of this discussion I use the term "bindep" for the bindep.txt but obviously it could mean improving the tool as well. There is a lot to encode: - build time / run time - this is already doable with tags and rather pretty clean, except for no linkage between the two concepts (which is what I tried to attack by "feature" concept) - applicable distro - rather supported - whether optional and when to use (think extras in setup.cfg) - rather supported, wonder if in the cleanest way, but it worked for oslo.cache gate to define functional tests based on backend Any others? we have a weird redhat downstream requiremetn that we were debating internally proposing as a change to upstream basicaly in our downstream ci we dont use pip or at least try not too. there are some optional dependices that are needed for optional backend not supproted in our product that do not have package, so those come form pip but otherwise we build an updated version of the distro package and use that.
so the idea our ci folks came up with was listign the proejct itself in bindeps. i found that to be quite strange and was kind fo arguing against that but if we had a convention where each project woudl list the name of the distro package for that project with a "self" tag and we ensureed it is only installed when that tag is used then that would make downstrema ci simpler e.g. the ci job would juse use zull to prepare all the repose and build the rpms then in the job we woudl do "bindep -l newline self | xargs <package mangager> install -y" that would pull in the newly built rpm for that project and use the package depencies in the distor to pull in the set of pacakges we expect. the issue we have today and what having a self tag would solve is we have to maintain a list of dependcies https://github.com/redhat-openstack/octario/blob/master/components_config/16... and then our downstream ci today truncate the requiremets.txt and test-requirements.txt so tehy are empty and run tox with --site-pacakges to use the depency we installed with the package manager. https://github.com/redhat-openstack/octario/blob/master/components_config/16... migi who maintains this part of our down stream ci has already done some experiments to use bindeps instead a moth or two ago wehre he propsoed a downstream patch to make a few small cahnges to the bindeps file to properly include all the deps aso i think this is something that would be useful outside of the container discuss too. i have cc'd migi who can explain this better and maybe separately share his expirence with using bindeps in this way as i have proably explained that terribly :)
-yoctozepto