[kolla] How to patch images during build
Folks,
I'm using kolla-build to build all images and push them to the local repo. So far all good but let's say If in future for some reason I want to patch some bug and rebuild the image in that case how do i patch kolla images?
I am reading at [1] and didn't see any example to patch Image for any bug.
Should I be downloading tarball and patch it and then use type=local to build the image from local source or add patch to Dockerfile? What people use for best practice for this kind of thing?
[1] https://docs.openstack.org/kolla/latest/admin/image-building.html
Hi Satish,
The correct way to apply bugfix is build the image from the git source. Commit bugfix changes to the git, and build from the bugfix branch for example.
On Wed, May 17, 2023 at 7:30 PM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'm using kolla-build to build all images and push them to the local repo. So far all good but let's say If in future for some reason I want to patch some bug and rebuild the image in that case how do i patch kolla images?
I am reading at [1] and didn't see any example to patch Image for any bug.
Should I be downloading tarball and patch it and then use type=local to build the image from local source or add patch to Dockerfile? What people use for best practice for this kind of thing?
[1] https://docs.openstack.org/kolla/latest/admin/image-building.html
Thank Maksim,
Like the following example? For reference, can I pass commit hash?
*[keystone-base]*
type = git
location = https://opendev.org/openstack/keystone
reference = stable/mitaka
On Wed, May 17, 2023 at 1:43 PM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
The correct way to apply bugfix is build the image from the git source. Commit bugfix changes to the git, and build from the bugfix branch for example.
On Wed, May 17, 2023 at 7:30 PM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'm using kolla-build to build all images and push them to the local repo. So far all good but let's say If in future for some reason I want to patch some bug and rebuild the image in that case how do i patch kolla images?
I am reading at [1] and didn't see any example to patch Image for any bug.
Should I be downloading tarball and patch it and then use type=local to build the image from local source or add patch to Dockerfile? What people use for best practice for this kind of thing?
[1] https://docs.openstack.org/kolla/latest/admin/image-building.html
-- Regards, Maksim Malchuk
Yes, you can do that, but note bene mitaka not supported.
On Wed, May 17, 2023 at 9:01 PM Satish Patel satish.txt@gmail.com wrote:
Thank Maksim,
Like the following example? For reference, can I pass commit hash?
*[keystone-base]*
type = git
location = https://opendev.org/openstack/keystone
reference = stable/mitaka
On Wed, May 17, 2023 at 1:43 PM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
The correct way to apply bugfix is build the image from the git source. Commit bugfix changes to the git, and build from the bugfix branch for example.
On Wed, May 17, 2023 at 7:30 PM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'm using kolla-build to build all images and push them to the local repo. So far all good but let's say If in future for some reason I want to patch some bug and rebuild the image in that case how do i patch kolla images?
I am reading at [1] and didn't see any example to patch Image for any bug.
Should I be downloading tarball and patch it and then use type=local to build the image from local source or add patch to Dockerfile? What people use for best practice for this kind of thing?
[1] https://docs.openstack.org/kolla/latest/admin/image-building.html
-- Regards, Maksim Malchuk
That was example, I took from here - https://docs.openstack.org/kolla/latest/admin/image-building.html
On Wed, May 17, 2023 at 2:02 PM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Yes, you can do that, but note bene mitaka not supported.
On Wed, May 17, 2023 at 9:01 PM Satish Patel satish.txt@gmail.com wrote:
Thank Maksim,
Like the following example? For reference, can I pass commit hash?
*[keystone-base]*
type = git
location = https://opendev.org/openstack/keystone
reference = stable/mitaka
On Wed, May 17, 2023 at 1:43 PM Maksim Malchuk maksim.malchuk@gmail.com wrote:
Hi Satish,
The correct way to apply bugfix is build the image from the git source. Commit bugfix changes to the git, and build from the bugfix branch for example.
On Wed, May 17, 2023 at 7:30 PM Satish Patel satish.txt@gmail.com wrote:
Folks,
I'm using kolla-build to build all images and push them to the local repo. So far all good but let's say If in future for some reason I want to patch some bug and rebuild the image in that case how do i patch kolla images?
I am reading at [1] and didn't see any example to patch Image for any bug.
Should I be downloading tarball and patch it and then use type=local to build the image from local source or add patch to Dockerfile? What people use for best practice for this kind of thing?
[1] https://docs.openstack.org/kolla/latest/admin/image-building.html
-- Regards, Maksim Malchuk
-- Regards, Maksim Malchuk
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion.
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
IMHO, patching - is not a production way.
On Tue, May 23, 2023 at 3:11 PM Michal Arbet michal.arbet@ultimum.io wrote:
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
Haha, really ? I don't know what to say ? Have you ever check deb or rpm packages ? And their patches ?
On Tue, May 23, 2023, 15:47 Maksim Malchuk maksim.malchuk@gmail.com wrote:
IMHO, patching - is not a production way.
On Tue, May 23, 2023 at 3:11 PM Michal Arbet michal.arbet@ultimum.io wrote:
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
-- Regards, Maksim Malchuk
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation. ________________________________ From: Michal Arbet michal.arbet@ultimum.io Sent: 23 May 2023 13:01 To: openstack-discuss@lists.openstack.org openstack-discuss@lists.openstack.org Subject: Re: [kolla] How to patch images during build
CAUTION: This email originates from outside THG
________________________________ I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.htmlhttps://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer [https://www.google.com/a/ultimum.io/images/logo.gif] Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.iomailto:michal.arbet@ultimum.io https://ultimum.iohttps://ultimum.io
LinkedInhttps://www.linkedin.com/company/ultimum-technologies | Twitterhttps://twitter.com/ultimumtech | Facebookhttps://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley <fungi@yuggoth.orgmailto:fungi@yuggoth.org> napsal: On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb Danny.Webb@thehutgroup.com wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 23 May 2023 13:01 *To:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts.
On Tue, May 23, 2023 at 7:47 PM Michal Arbet michal.arbet@ultimum.io wrote:
How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb Danny.Webb@thehutgroup.com wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 23 May 2023 13:01 *To:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
The correct and only way to apply patches on images - is build them from source. You should patch the code, not binary files.
On Wed, May 24, 2023 at 3:32 AM Satish Patel satish.txt@gmail.com wrote:
This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts.
On Tue, May 23, 2023 at 7:47 PM Michal Arbet michal.arbet@ultimum.io wrote:
How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb Danny.Webb@thehutgroup.com wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 23 May 2023 13:01 *To:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
Okay, let's say you found a critical bug and you have already sent a patch for review. This - as you know can take quite a long time - gate broken, unit tests are not working etc etc but this is still regular fix and merged in other versions. Moreover upstream guys need to release a new pip package and amend upper constraints.
But you need to fix your production now ..how can you do it in kolla now ? You cannot.
As I said before as an example .. Oslo messaging is broken from xena to antelope, if your rabbitmq will go down on this versions ..your clients will not connect again to cluster - this is critical bug. So, let's check how upstream guys dealing with it :
1, Patch applied upstream ? - master - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/866617 - antelope - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/883533 - zed - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883537 - xena - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883539 - yoga - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883538
2, Okay, antelope merged , is the new version released as this is a critical bug ? - no , they again don't care -
* 0602d1a1 (HEAD -> master, origin/master, origin/HEAD) Increase ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay (Andrew Bogott, 5 weeks ago - 2023-04-20 15:27:58 -0500) * fd2381c7 (tag: 14.3.0) Disable greenthreads for RabbitDriver "listen" connections (Arnaud Morin, 3 months ago - 2023-03-03 11:24:27 +0100)
Last version is 14.3.0 and fix is still not released in pypi repo.
Other versions ? check 1.
Let's check how ubuntu handled this problem :
python-oslo.messaging (12.13.0-0ubuntu1.1) jammy; urgency=medium
* d/gbp.conf: Create stable/yoga branch. * d/p/revert-limit-maximum-timeout-in-the-poll-loop.patch: This reverts an upstream patch that is preventing active/active rabbitmq from failing over when a node goes down (LP: #1993149).
-- Corey Bryant corey.bryant@canonical.com Thu, 20 Oct 2022 15:48:16 -0400
They patched the buggy version !! Kolla dropped binary builds ...so you can't install dependencies from apt repository where it is patched, and you don't have a way how to patch
your python library.
Patching is normal way how to fix a problem, you don't have always option to bump version, you need patch code and kolla just don't have this option.
-
On Wed, May 24, 2023, 08:41 Maksim Malchuk maksim.malchuk@gmail.com wrote:
The correct and only way to apply patches on images - is build them from source. You should patch the code, not binary files.
On Wed, May 24, 2023 at 3:32 AM Satish Patel satish.txt@gmail.com wrote:
This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts.
On Tue, May 23, 2023 at 7:47 PM Michal Arbet michal.arbet@ultimum.io wrote:
How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb Danny.Webb@thehutgroup.com wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 23 May 2023 13:01 *To:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
-- Regards, Maksim Malchuk
You do it with kolla image builder, all of the dockerfiles are jinja and you can override or add in new blocks (such as patching a file, changing to different versions of a package etc). Eg:
https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2
So what we have is a set of pipelines that create patches based on upstream openstack repos, and then we apply these patches in our image build pipelines by changing files in image build process. This way you have immutable containers that you aren't trying to change after the fact in the kolla-ansible process. ________________________________ From: Michal Arbet michal.arbet@ultimum.io Sent: 24 May 2023 14:45 To: Maksim Malchuk maksim.malchuk@gmail.com Cc: Satish Patel satish.txt@gmail.com; Danny Webb Danny.Webb@thehutgroup.com; openstack-discuss openstack-discuss@lists.openstack.org Subject: Re: [kolla] How to patch images during build
CAUTION: This email originates from outside THG
________________________________ Okay, let's say you found a critical bug and you have already sent a patch for review. This - as you know can take quite a long time - gate broken, unit tests are not working etc etc but this is still regular fix and merged in other versions. Moreover upstream guys need to release a new pip package and amend upper constraints.
But you need to fix your production now ..how can you do it in kolla now ? You cannot.
As I said before as an example .. Oslo messaging is broken from xena to antelope, if your rabbitmq will go down on this versions ..your clients will not connect again to cluster - this is critical bug. So, let's check how upstream guys dealing with it :
1, Patch applied upstream ? - master - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/866617https://review.opendev.org/c/openstack/oslo.messaging/+/866617 - antelope - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/883533https://review.opendev.org/c/openstack/oslo.messaging/+/883533 - zed - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883537https://review.opendev.org/c/openstack/oslo.messaging/+/883537 - xena - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883539https://review.opendev.org/c/openstack/oslo.messaging/+/883539 - yoga - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883538https://review.opendev.org/c/openstack/oslo.messaging/+/883538
2, Okay, antelope merged , is the new version released as this is a critical bug ? - no , they again don't care -
* 0602d1a1 (HEAD -> master, origin/master, origin/HEAD) Increase ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay (Andrew Bogott, 5 weeks ago - 2023-04-20 15:27:58 -0500) * fd2381c7 (tag: 14.3.0) Disable greenthreads for RabbitDriver "listen" connections (Arnaud Morin, 3 months ago - 2023-03-03 11:24:27 +0100)
Last version is 14.3.0 and fix is still not released in pypi repo.
Other versions ? check 1.
Let's check how ubuntu handled this problem :
python-oslo.messaging (12.13.0-0ubuntu1.1) jammy; urgency=medium
* d/gbp.conf: Create stable/yoga branch. * d/p/revert-limit-maximum-timeout-in-the-poll-loop.patch: This reverts an upstream patch that is preventing active/active rabbitmq from failing over when a node goes down (LP: #1993149).
-- Corey Bryant <corey.bryant@canonical.commailto:corey.bryant@canonical.com> Thu, 20 Oct 2022 15:48:16 -0400
They patched the buggy version !! Kolla dropped binary builds ...so you can't install dependencies from apt repository where it is patched, and you don't have a way how to patch
your python library.
Patching is normal way how to fix a problem, you don't have always option to bump version, you need patch code and kolla just don't have this option.
-
On Wed, May 24, 2023, 08:41 Maksim Malchuk <maksim.malchuk@gmail.commailto:maksim.malchuk@gmail.com> wrote: The correct and only way to apply patches on images - is build them from source. You should patch the code, not binary files.
On Wed, May 24, 2023 at 3:32 AM Satish Patel <satish.txt@gmail.commailto:satish.txt@gmail.com> wrote: This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts.
On Tue, May 23, 2023 at 7:47 PM Michal Arbet <michal.arbet@ultimum.iomailto:michal.arbet@ultimum.io> wrote: How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb <Danny.Webb@thehutgroup.commailto:Danny.Webb@thehutgroup.com> wrote: You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation. ________________________________ From: Michal Arbet <michal.arbet@ultimum.iomailto:michal.arbet@ultimum.io> Sent: 23 May 2023 13:01 To: openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org <openstack-discuss@lists.openstack.orgmailto:openstack-discuss@lists.openstack.org> Subject: Re: [kolla] How to patch images during build
CAUTION: This email originates from outside THG
________________________________ I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.htmlhttps://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer [https://www.google.com/a/ultimum.io/images/logo.gif] Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.iomailto:michal.arbet@ultimum.io https://ultimum.iohttps://ultimum.io
LinkedInhttps://www.linkedin.com/company/ultimum-technologies | Twitterhttps://twitter.com/ultimumtech | Facebookhttps://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley <fungi@yuggoth.orgmailto:fungi@yuggoth.org> napsal: On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
-- Regards, Maksim Malchuk
Thank you michal for explaining the pain. I was also thinking how to put those things together for smooth rollout.
@Danny - We are running a yoga release that means the Oslo bugfix hasn't merged yet. In the above example how do I patch my images and which images that patch will go?
Yoga oslo bug - https://review.opendev.org/c/openstack/oslo.messaging/+/883538
Do you have a snippet for Dockerfile.j2 to apply to the above patch? Just looking for sample :)
On Wed, May 24, 2023 at 9:53 AM Danny Webb Danny.Webb@thehutgroup.com wrote:
You do it with kolla image builder, all of the dockerfiles are jinja and you can override or add in new blocks (such as patching a file, changing to different versions of a package etc). Eg:
https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2
So what we have is a set of pipelines that create patches based on upstream openstack repos, and then we apply these patches in our image build pipelines by changing files in image build process. This way you have immutable containers that you aren't trying to change after the fact in the kolla-ansible process.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 24 May 2023 14:45 *To:* Maksim Malchuk maksim.malchuk@gmail.com *Cc:* Satish Patel satish.txt@gmail.com; Danny Webb < Danny.Webb@thehutgroup.com>; openstack-discuss < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
Okay, let's say you found a critical bug and you have already sent a patch for review. This - as you know can take quite a long time - gate broken, unit tests are not working etc etc but this is still regular fix and merged in other versions. Moreover upstream guys need to release a new pip package and amend upper constraints.
But you need to fix your production now ..how can you do it in kolla now ? You cannot.
As I said before as an example .. Oslo messaging is broken from xena to antelope, if your rabbitmq will go down on this versions ..your clients will not connect again to cluster - this is critical bug. So, let's check how upstream guys dealing with it :
1, Patch applied upstream ? - master - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/866617 - antelope - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/883533 - zed - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883537 - xena - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883539 - yoga - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883538
2, Okay, antelope merged , is the new version released as this is a critical bug ? - no , they again don't care -
- 0602d1a1 (HEAD -> master, origin/master, origin/HEAD) Increase
ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay (Andrew Bogott, 5 weeks ago - 2023-04-20 15:27:58 -0500)
- fd2381c7 (tag: 14.3.0) Disable greenthreads for RabbitDriver "listen"
connections (Arnaud Morin, 3 months ago - 2023-03-03 11:24:27 +0100)
Last version is 14.3.0 and fix is still not released in pypi repo.
Other versions ? check 1.
Let's check how ubuntu handled this problem :
python-oslo.messaging (12.13.0-0ubuntu1.1) jammy; urgency=medium
- d/gbp.conf: Create stable/yoga branch.
- d/p/revert-limit-maximum-timeout-in-the-poll-loop.patch: This reverts an upstream patch that is preventing active/active rabbitmq from failing over when a node goes down (LP: #1993149).
-- Corey Bryant corey.bryant@canonical.com Thu, 20 Oct 2022 15:48:16 -0400
They patched the buggy version !! Kolla dropped binary builds ...so you can't install dependencies from apt repository where it is patched, and you don't have a way how to patch
your python library.
Patching is normal way how to fix a problem, you don't have always option to bump version, you need patch code and kolla just don't have this option.
-On Wed, May 24, 2023, 08:41 Maksim Malchuk maksim.malchuk@gmail.com wrote:
The correct and only way to apply patches on images - is build them from source. You should patch the code, not binary files.
On Wed, May 24, 2023 at 3:32 AM Satish Patel satish.txt@gmail.com wrote:
This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts.
On Tue, May 23, 2023 at 7:47 PM Michal Arbet michal.arbet@ultimum.io wrote:
How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb Danny.Webb@thehutgroup.com wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 23 May 2023 13:01 *To:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
-- Regards, Maksim Malchuk
I know that there is that option. I just think that there should be a simpler option to patch. Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 24. 5. 2023 v 15:53 odesílatel Danny Webb Danny.Webb@thehutgroup.com napsal:
You do it with kolla image builder, all of the dockerfiles are jinja and you can override or add in new blocks (such as patching a file, changing to different versions of a package etc). Eg:
https://github.com/openstack/kolla/blob/master/docker/rabbitmq/Dockerfile.j2
So what we have is a set of pipelines that create patches based on upstream openstack repos, and then we apply these patches in our image build pipelines by changing files in image build process. This way you have immutable containers that you aren't trying to change after the fact in the kolla-ansible process.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 24 May 2023 14:45 *To:* Maksim Malchuk maksim.malchuk@gmail.com *Cc:* Satish Patel satish.txt@gmail.com; Danny Webb < Danny.Webb@thehutgroup.com>; openstack-discuss < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
Okay, let's say you found a critical bug and you have already sent a patch for review. This - as you know can take quite a long time - gate broken, unit tests are not working etc etc but this is still regular fix and merged in other versions. Moreover upstream guys need to release a new pip package and amend upper constraints.
But you need to fix your production now ..how can you do it in kolla now ? You cannot.
As I said before as an example .. Oslo messaging is broken from xena to antelope, if your rabbitmq will go down on this versions ..your clients will not connect again to cluster - this is critical bug. So, let's check how upstream guys dealing with it :
1, Patch applied upstream ? - master - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/866617 - antelope - OK merged - https://review.opendev.org/c/openstack/oslo.messaging/+/883533 - zed - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883537 - xena - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883539 - yoga - they don't care - https://review.opendev.org/c/openstack/oslo.messaging/+/883538
2, Okay, antelope merged , is the new version released as this is a critical bug ? - no , they again don't care -
- 0602d1a1 (HEAD -> master, origin/master, origin/HEAD) Increase
ACK_REQUEUE_EVERY_SECONDS_MAX to exceed default kombu_reconnect_delay (Andrew Bogott, 5 weeks ago - 2023-04-20 15:27:58 -0500)
- fd2381c7 (tag: 14.3.0) Disable greenthreads for RabbitDriver "listen"
connections (Arnaud Morin, 3 months ago - 2023-03-03 11:24:27 +0100)
Last version is 14.3.0 and fix is still not released in pypi repo.
Other versions ? check 1.
Let's check how ubuntu handled this problem :
python-oslo.messaging (12.13.0-0ubuntu1.1) jammy; urgency=medium
- d/gbp.conf: Create stable/yoga branch.
- d/p/revert-limit-maximum-timeout-in-the-poll-loop.patch: This reverts an upstream patch that is preventing active/active rabbitmq from failing over when a node goes down (LP: #1993149).
-- Corey Bryant corey.bryant@canonical.com Thu, 20 Oct 2022 15:48:16 -0400
They patched the buggy version !! Kolla dropped binary builds ...so you can't install dependencies from apt repository where it is patched, and you don't have a way how to patch
your python library.
Patching is normal way how to fix a problem, you don't have always option to bump version, you need patch code and kolla just don't have this option.
-On Wed, May 24, 2023, 08:41 Maksim Malchuk maksim.malchuk@gmail.com wrote:
The correct and only way to apply patches on images - is build them from source. You should patch the code, not binary files.
On Wed, May 24, 2023 at 3:32 AM Satish Patel satish.txt@gmail.com wrote:
This is very interesting that there is no standard or best way to patch images. Everyone uses their own way to handle it. Now I am very curious to see how folks running it in production with patching and building images. I am about to deploy kolla on production and trying to learn all best practices from experts.
On Tue, May 23, 2023 at 7:47 PM Michal Arbet michal.arbet@ultimum.io wrote:
How ?
Say to me please ? How you can patch oslo.messaging except template override
On Tue, May 23, 2023, 16:17 Danny Webb Danny.Webb@thehutgroup.com wrote:
You can already do this with the kolla image builder which seems to me to be a much better solution than patching containers post creation.
*From:* Michal Arbet michal.arbet@ultimum.io *Sent:* 23 May 2023 13:01 *To:* openstack-discuss@lists.openstack.org < openstack-discuss@lists.openstack.org> *Subject:* Re: [kolla] How to patch images during build
- CAUTION: This email originates from outside THG *
I am very glad that someone asked for an option to patch kolla images. I've already proposed patches for kolla here [1] and here [2]. But unfortunately I didn't get that many votes to merge into master and I abandoned this.
[1] https://review.opendev.org/c/openstack/kolla/+/829296 [2] https://review.opendev.org/c/openstack/kolla/+/829295
With these above patches you can patch files inside every container. Maybe we can discuss this again ??
For example now xena, yoga, zed, antelope has oslo.messaging broken :
https://bugs.launchpad.net/oslo.messaging/+bug/2019978 fixed by https://review.opendev.org/c/openstack/oslo.messaging/+/866617
As I am using my kolla patches in my downstream kolla git repo i've only created patches/ directory and place fix for openstack-base container :)
patches/ patches/openstack-base patches/openstack-base/series patches/openstack-base/fix-rabbitmq-issue-opendev-883538.patch
But, you still can use template-override https://docs.openstack.org/kolla/latest/admin/image-building.html .
Thanks
Michal Arbet Openstack Engineer
Ultimum Technologies a.s. Na Poříčí 1047/26, 11000 Praha 1 Czech Republic
+420 604 228 897 michal.arbet@ultimum.io *https://ultimum.io https://ultimum.io/*
LinkedIn https://www.linkedin.com/company/ultimum-technologies | Twitter https://twitter.com/ultimumtech | Facebook https://www.facebook.com/ultimumtechnologies/timeline
st 17. 5. 2023 v 20:19 odesílatel Jeremy Stanley fungi@yuggoth.org napsal:
On 2023-05-17 21:02:02 +0300 (+0300), Maksim Malchuk wrote:
Yes, you can do that, but note bene mitaka not supported.
[...]
Not only unsupported, but the stable/mitaka branch of openstack/keystone was deleted when it reached EOL in 2017. You may instead want to specify `reference = mitaka-eol` (assuming Git tags also work there). That should get you the final state of the stable/mitaka branch prior to its deletion. -- Jeremy Stanley
-- Regards, Maksim Malchuk
participants (5)
-
Danny Webb
-
Jeremy Stanley
-
Maksim Malchuk
-
Michal Arbet
-
Satish Patel