Open Stack

On Wed, 2020-04-22 at 15:24 +0200, Radosław Piliszek wrote:
> On Tue, Apr 21, 2020 at 9:12 PM Mohammed Naser <mnaser at vexxhost.com> wrote:
> > 
> > On Tue, Apr 21, 2020 at 5:15 AM Mark Goddard <mark at stackhpc.com> wrote:
> > > 
> > > On Sat, 18 Apr 2020 at 15:10, Radosław Piliszek
> > > <radoslaw.piliszek at gmail.com> wrote:
> > > > 
> > > > The above effort could well be coordinated with different projects to
> > > > reuse bindep contents. So far Kolla does not use bindep because it
> > > > often installs too much and not enough at the same time.
> > > > Do note it would still be bindep-less for external services.
> > > 
> > > I like the idea of using bindep for source images, in combination with
> > > multi-stage builds. If we could reduce the length of our package
> > > dependency lists by relying on bindep, that would be a nice win.
> > > Ideally, between distro package dependencies and bindep, we would need
> > > to specify few additional packages.
> > 
> > I think this is a golden ticket of what I'm trying to accomplish and this is
> > what OpenDev is currently doing.  Ideally, there _should_ be no distro package
> > dependencies because they should all be modeled inside bindep (using different
> > profiles if needed).
> > 
> > I still think that if we do this, there is a lot of value in being
> > able to do this in-repo
> > because those Dockerfile's will be pretty static.  While I appreciate
> > that Kolla has
> > all sorts of neat things to customize images, for a certain operator
> > base, they'd
> > much rather either have a Dockerfile to build off of or an existing base image
> > somewhere.
> > 
> > Kolla could be the thing that drives this, Kolla could become
> > "OpenStack containers",
> > or parts of it could be.  However, I think the "customization bits"
> > should ideally stay
> > outside of the scope of this, because we just want to publish an artifact with
> > dependencies installed.
> 
> I think we all agree here that bindep is the way to go and whatever we
> decide on should include "better bindep experience".
> Could we collect together some concrete ideas how to make the bindep
> better? (and then suddenly bindep sig appears out of nowhere :O )
> I guess Kolla is a great source especially for optional runtime bindep
> entries, though it can really only offer names of packages per distro
> rather than insight why they are used (except for git archaelogy of
> course).
> We could collaborate with each project to discover real relations between these.
> Making bindep better would benefit all efforts of deployment (with
> devstack included) so it's a win-win.
> 
> For the purpose of this discussion I use the term "bindep" for the
> bindep.txt but obviously it could mean improving the tool as well.
> There is a lot to encode:
> - build time / run time - this is already doable with tags and rather
> pretty clean, except for no linkage between the two concepts (which is
> what I tried to attack by "feature" concept)
> - applicable distro - rather supported
> - whether optional and when to use (think extras in setup.cfg) -
> rather supported, wonder if in the cleanest way, but it worked for
> oslo.cache gate to define functional tests based on backend
> Any others?
we have a weird redhat downstream requiremetn that we were debating internally proposing as a change to upstream
basicaly in our downstream ci we dont use pip or at least try not too. there are some optional dependices that are
needed for optional backend not supproted in our product that do not have package, so those come form pip but otherwise
we build an updated version  of the distro package and use that.

so the idea our ci folks came up with was listign the proejct itself in bindeps.
i found that to be quite strange and was kind fo arguing against that but if we had a convention where each project
woudl list the name of the distro package for that project with a "self" tag and we ensureed it is only installed when
that tag is used then that would make downstrema ci simpler

e.g. the ci job would juse use zull to prepare all the repose and build the rpms then in the job we woudl do

"bindep -l newline self | xargs <package mangager> install -y" that would pull in the newly built rpm for that project
and use the package depencies in the distor to pull in the set of pacakges we expect. the issue we have today and what
having a self tag would solve is we have to maintain a list of dependcies 
https://github.com/redhat-openstack/octario/blob/master/components_config/16.1/nova/component.yml#L26-L60
and then  our downstream ci today truncate the requiremets.txt and test-requirements.txt so tehy are empty and run tox
with --site-pacakges to use the depency we installed with the package manager. 
https://github.com/redhat-openstack/octario/blob/master/components_config/16.1/nova/component.yml#L159-L187

migi who maintains this part of our down stream ci has already done some experiments to use bindeps instead a moth or
two ago wehre he propsoed a downstream patch to make a few small cahnges to the bindeps file to properly include all the
deps aso i think this is something that would be useful outside of the container discuss too.
i have cc'd migi who can explain this better and maybe separately share his expirence with using bindeps in this way
as i have proably explained that terribly :)


> 
> -yoctozepto
>