You do it with kolla image builder, all of the dockerfiles are jinja and you can override or add in new blocks (such as patching a file, changing to different versions of a package etc).  Eg:

https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2

So what we have is a set of pipelines that create patches based on upstream openstack repos, and then we apply these patches in our image build pipelines by changing files in image build process.  This way you have immutable containers that you aren't trying to change after the fact in the kolla-ansible process.  
From: Michal Arbet <michal.arbet@ultimum.io>
Sent: 24 May 2023 14:45
To: Maksim Malchuk <maksim.malchuk@gmail.com>
Cc: Satish Patel <satish.txt@gmail.com>; Danny Webb <Danny.Webb@thehutgroup.com>; openstack-discuss <openstack-discuss@lists.openstack.org>
Subject: Re: [kolla] How to patch images during build
 

CAUTION: This email originates from outside THG

Okay, let's say you found a critical bug and you have already sent a patch for review.
 This - as you know can take quite a long time - gate broken, unit tests are not working etc etc but this is still regular fix and merged in other versions. 
Moreover upstream guys need to release a new pip package and amend upper constraints.

But you need to fix your production now ..how can you do it in kolla now ? You cannot. 

As I said before as an example .. Oslo messaging is broken from xena to antelope, if your rabbitmq will go down on this versions ..your clients will not connect again to cluster - this is critical bug.
So, let's check how upstream guys dealing with it : 

1, Patch applied upstream ? 
    - master - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/866617
    - antelope - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/883533
    - zed - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883537
    - xena - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883539
    - yoga - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883538

2, Okay, antelope merged , is the new version released as this is a critical bug ?
    - no , they again don't care -

 * 0602d1a1 (HEAD -> master, origin/master, origin/HEAD) Increase ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay (Andrew Bogott, 5 weeks ago - 2023-04-20 15:27:58 -0500)
* fd2381c7 (tag: 14.3.0) Disable greenthreads for RabbitDriver "listen" connections (Arnaud Morin, 3 months ago - 2023-03-03 11:24:27 +0100)

Last version is 14.3.0 and fix is still not released in pypi repo.

   Other versions ? check 1.

Let's check how ubuntu handled this problem : 

python-oslo.messaging (12.13.0-0ubuntu1.1) jammy; urgency=medium

  * d/gbp.conf: Create stable/yoga branch.
  * d/p/revert-limit-maximum-timeout-in-the-poll-loop.patch: This reverts
    an upstream patch that is preventing active/active rabbitmq from
    failing over when a node goes down (LP: #1993149).

 -- Corey Bryant <corey.bryant@canonical.com>  Thu, 20 Oct 2022 15:48:16 -0400

They patched the buggy version !! Kolla dropped binary builds ...so you can't install dependencies from apt repository where it is patched, and you don't have a way how to patch 
your python library.

Patching is normal way how to fix a problem, you don't have always option to bump version, you need patch code and kolla just don't have this option.



    - 

On Wed, May 24, 2023, 08:41 Maksim Malchuk <maksim.malchuk@gmail.com> wrote:
The correct and only way to apply patches on images - is build them from source. You should patch the code, not binary files.

On Wed, May 24, 2023 at 3:32 AM Satish Patel <satish.txt@gmail.com> wrote:
This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts. 

On Tue, May 23, 2023 at 7:47 PM Michal Arbet <michal.arbet@ultimum.io> wrote:
How ? 

Say to me please ? How you can patch oslo.messaging except template override 

On Tue, May 23, 2023, 16:17 Danny Webb <Danny.Webb@thehutgroup.com> wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.  
From: Michal Arbet <michal.arbet@ultimum.io>
Sent: 23 May 2023 13:01
To: openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.org>
Subject: Re: [kolla] How to patch images during build
 

CAUTION: This email originates from outside THG

I am very glad that someone asked for an option to patch kolla images. 
I've already proposed patches for kolla here [1] and here [2]. 
But unfortunately I didn't get that many votes to merge into master and I abandoned this.


With these above patches you can patch files inside every container.
Maybe we can discuss this again ??

For example now xena, yoga, zed, antelope has oslo.messaging broken :

fixed by

As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)

patches/
patches/openstack-base
patches/openstack-base/series
patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch

But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .

Thanks

Michal Arbet
Openstack Engineer

Ultimum Technologies a.s.
Na Poříčí 1047/26, 11000 Praha 1
Czech Republic

+420 604 228 897 
michal.arbet@ultimum.io
https://ultimum.io



st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley <fungi@yuggoth.org> napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
> Yes, you can do that, but note bene mitaka not supported.
[...]

Not only unsupported, but the stable/mitaka branch of
openstack/keystone was deleted when it reached EOL in 2017. You may
instead want to specify `reference = mitaka-eol` (assuming Git tags
also work there). That should get you the final state of the
stable/mitaka branch prior to its deletion.
--
Jeremy Stanley


--
Regards,
Maksim Malchuk