Open Stack

Thank you michal for explaining the pain. I was also thinking how to put
those things together for smooth rollout.

@Danny - We are running a yoga release that means the Oslo bugfix hasn't
merged yet. In the above example how do I patch my images and which images
that patch will go?

 Yoga oslo bug -
https://review.opendev.org/c/openstack/oslo.messaging/+/883538

Do you have a snippet for Dockerfile.j2 to apply to the above patch? Just
looking for sample :)

On Wed, May 24, 2023 at 9:53 AM Danny Webb <Danny.Webb at thehutgroup.com>
wrote:

> You do it with kolla image builder, all of the dockerfiles are jinja and
> you can override or add in new blocks (such as patching a file, changing to
> different versions of a package etc).  Eg:
>
>
> https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2
>
> So what we have is a set of pipelines that create patches based on
> upstream openstack repos, and then we apply these patches in our image
> build pipelines by changing files in image build process.  This way you
> have immutable containers that you aren't trying to change after the fact
> in the kolla-ansible process.
> ------------------------------
> *From:* Michal Arbet <michal.arbet at ultimum.io>
> *Sent:* 24 May 2023 14:45
> *To:* Maksim Malchuk <maksim.malchuk at gmail.com>
> *Cc:* Satish Patel <satish.txt at gmail.com>; Danny Webb <
> Danny.Webb at thehutgroup.com>; openstack-discuss <
> openstack-discuss at lists.openstack.org>
> *Subject:* Re: [kolla] How to patch images during build
>
>
> * CAUTION: This email originates from outside THG *
> ------------------------------
> Okay, let's say you found a critical bug and you have already sent a patch
> for review.
>  This - as you know can take quite a long time - gate broken, unit tests
> are not working etc etc but this is still regular fix and merged in other
> versions.
> Moreover upstream guys need to release a new pip package and amend upper
> constraints.
>
> But you need to fix your production now ..how can you do it in kolla now ?
> You cannot.
>
> As I said before as an example .. Oslo messaging is broken from xena to
> antelope, if your rabbitmq will go down on this versions ..your clients
> will not connect again to cluster - this is critical bug.
> So, let's check how upstream guys dealing with it :
>
> 1, Patch applied upstream ?
>     - master - OK merged -
> https://review.opendev.org/c/openstack/oslo.messaging/+/866617
>     - antelope - OK merged -
> https://review.opendev.org/c/openstack/oslo.messaging/+/883533
>     - zed - they don't care -
> https://review.opendev.org/c/openstack/oslo.messaging/+/883537
>     - xena - they don't care -
> https://review.opendev.org/c/openstack/oslo.messaging/+/883539
>     - yoga - they don't care -
> https://review.opendev.org/c/openstack/oslo.messaging/+/883538
>
> 2, Okay, antelope merged , is the new version released as this is a
> critical bug ?
>     - no , they again don't care -
>
>  * 0602d1a1 (HEAD -> master, origin/master, origin/HEAD) Increase
> ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay
> (Andrew Bogott, 5 weeks ago - 2023-04-20 15:27:58 -0500)
> * fd2381c7 (tag: 14.3.0) Disable greenthreads for RabbitDriver "listen"
> connections (Arnaud Morin, 3 months ago - 2023-03-03 11:24:27 +0100)
>
> Last version is 14.3.0 and fix is still not released in pypi repo.
>
>    Other versions ? check 1.
>
> Let's check how ubuntu handled this problem :
>
> python-oslo.messaging (12.13.0-0ubuntu1.1) jammy; urgency=medium
>
>   * d/gbp.conf: Create stable/yoga branch.
>   * d/p/revert-limit-maximum-timeout-in-the-poll-loop.patch: This reverts
>     an upstream patch that is preventing active/active rabbitmq from
>     failing over when a node goes down (LP: #1993149).
>
>  -- Corey Bryant <corey.bryant at canonical.com>  Thu, 20 Oct 2022 15:48:16 -0400
>
>
> They patched the buggy version !! Kolla dropped binary builds ...so you can't install dependencies from apt repository where it is patched, and you don't have a way how to patch
>
> your python library.
>
>
> Patching is normal way how to fix a problem, you don't have always option to bump version, you need patch code and kolla just don't have this option.
>
>
>
>
>     -
>
> On Wed, May 24, 2023, 08:41 Maksim Malchuk <maksim.malchuk at gmail.com>
> wrote:
>
> The correct and only way to apply patches on images - is build them from
> source. You should patch the code, not binary files.
>
> On Wed, May 24, 2023 at 3:32 AM Satish Patel <satish.txt at gmail.com> wrote:
>
> This is very interesting that there is no standard or best way to patch
> images. Everyone uses their own way to handle it. Now I am very curious to
> see how folks running it in production with patching and building images. I
> am about to deploy kolla on production and trying to learn all best
> practices from experts.
>
> On Tue, May 23, 2023 at 7:47 PM Michal Arbet <michal.arbet at ultimum.io>
> wrote:
>
> How ?
>
> Say to me please ? How you can patch oslo.messaging except template
> override
>
> On Tue, May 23, 2023, 16:17 Danny Webb <Danny.Webb at thehutgroup.com> wrote:
>
> You can already do this with the kolla image builder which seems to me to
> be a much better solution than patching containers post creation.
> ------------------------------
> *From:* Michal Arbet <michal.arbet at ultimum.io>
> *Sent:* 23 May 2023 13:01
> *To:* openstack-discuss at lists.openstack.org <
> openstack-discuss at lists.openstack.org>
> *Subject:* Re: [kolla] How to patch images during build
>
>
> * CAUTION: This email originates from outside THG *
> ------------------------------
> I am very glad that someone asked for an option to patch kolla images.
> I've already proposed patches for kolla here [1] and here [2].
> But unfortunately I didn't get that many votes to merge into master and I
> abandoned this.
>
> [1] https://review.opendev.org/c/openstack/kolla/+/829296
> [2] https://review.opendev.org/c/openstack/kolla/+/829295
>
> With these above patches you can patch files inside every container.
> Maybe we can discuss this again ??
>
> For example now xena, yoga, zed, antelope has oslo.messaging broken :
>
> https://bugs.launchpad.net/oslo.messaging/+bug/2019978
> fixed by
> https://review.opendev.org/c/openstack/oslo.messaging/+/866617
>
> As I am using my kolla patches in my downstream kolla git repo i've only
> created patches/ directory and place fix for openstack-base container :)
>
> patches/
> patches/openstack-base
> patches/openstack-base/series
> patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
>
> But, you still can use template-override
> https://docs.openstack.org/kolla/latest/admin/image-building.html .
>
> Thanks
>
> Michal Arbet
> Openstack Engineer
>
> Ultimum Technologies a.s.
> Na Poříčí 1047/26, 11000 Praha 1
> Czech Republic
>
> +420 604 228 897
> michal.arbet at ultimum.io
> *https://ultimum.io <https://ultimum.io/>*
>
> LinkedIn <https://www.linkedin.com/company/ultimum-technologies> | Twitter
> <https://twitter.com/ultimumtech> | Facebook
> <https://www.facebook.com/ultimumtechnologies/timeline>
>
>
> st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley <fungi at yuggoth.org>
> napsal:
>
> On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
> > Yes, you can do that, but note bene mitaka not supported.
> [...]
>
> Not only unsupported, but the stable/mitaka branch of
> openstack/keystone was deleted when it reached EOL in 2017. You may
> instead want to specify `reference = mitaka-eol` (assuming Git tags
> also work there). That should get you the final state of the
> stable/mitaka branch prior to its deletion.
> --
> Jeremy Stanley
>
>
>
> --
> Regards,
> Maksim Malchuk
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230524/527a2c09/attachment.htm>