[tripleo] Use Podman 1.6 in CI
Hey folks, We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. A possible workaround (originally created by Damien) can be found here [1]. A few of us been in talks with the Podman team about this problem and have tested with a newer version of it (1.6.4, to be exact) and found that it is no longer an issue. The most ideal situation is to simply use this newer version of Podman instead of adding hacky workarounds that we will soon revert. However, I am unsure about how we would go about doing this upstream. RHEL will soon get an updated Podman version but CentOS always lags behind. Even CentOS 8 Stream does not contain the newer version [2]. The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? 1. https://review.opendev.org/#/c/698999/ 2. http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/ Sincerely, Luke Short
On 2020-01-08 16:49:15 -0500 (-0500), Luke Short wrote:
We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. [...] The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? [...]
This sounds like a problem users of your software could encounter in production. If so, how does only fixing it in CI jobs help your users? It seems like time might be better spent fixing the problem for everyone. -- Jeremy Stanley
On Wed, Jan 8, 2020 at 3:30 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-08 16:49:15 -0500 (-0500), Luke Short wrote:
We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. [...] The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? [...]
This sounds like a problem users of your software could encounter in production. If so, how does only fixing it in CI jobs help your users? It seems like time might be better spent fixing the problem for everyone.
Btw fixing CI implies fixing for everyone. In other words, how do we make it available for everyone (including CI). This is one of those ecosystem things because we (tripleo/openstack) don't necessarily ship it but we do need to use it. I'm uncertain of the centos7/podman 1.6 support and which branches are affected by this? This might be a better question for RDO. Thanks, -Alex
-- Jeremy Stanley
On 2020-01-08 15:35:03 -0700 (-0700), Alex Schultz wrote:
On Wed, Jan 8, 2020 at 3:30 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-08 16:49:15 -0500 (-0500), Luke Short wrote:
We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. [...] The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? [...]
This sounds like a problem users of your software could encounter in production. If so, how does only fixing it in CI jobs help your users? It seems like time might be better spent fixing the problem for everyone.
Btw fixing CI implies fixing for everyone. In other words, how do we make it available for everyone (including CI). This is one of those ecosystem things because we (tripleo/openstack) don't necessarily ship it but we do need to use it. I'm uncertain of the centos7/podman 1.6 support and which branches are affected by this? This might be a better question for RDO.
I see, "ship/use a newer version of Podman in our upstream CI" didn't seem to necessarily imply getting a newer version of Podman into RDO/TripleO and the hands of its users. I have a bit of a knee-jerk reaction whenever I see someone talk about "fixing CI" when the underlying problem is in the software being tested and not the CI jobs. -- Jeremy Stanley
Hey folks, Thank you for all of the feedback so far. The goal is definitely to fix this everywhere we can, no just in CI. Sorry for my poor choice of words. I will migrate this discussion over to the RDO community. Sincerely, Luke Short On Wed, Jan 8, 2020 at 5:55 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-08 15:35:03 -0700 (-0700), Alex Schultz wrote:
On Wed, Jan 8, 2020 at 3:30 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-08 16:49:15 -0500 (-0500), Luke Short wrote:
We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. [...] The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? [...]
This sounds like a problem users of your software could encounter in production. If so, how does only fixing it in CI jobs help your users? It seems like time might be better spent fixing the problem for everyone.
Btw fixing CI implies fixing for everyone. In other words, how do we make it available for everyone (including CI). This is one of those ecosystem things because we (tripleo/openstack) don't necessarily ship it but we do need to use it. I'm uncertain of the centos7/podman 1.6 support and which branches are affected by this? This might be a better question for RDO.
I see, "ship/use a newer version of Podman in our upstream CI" didn't seem to necessarily imply getting a newer version of Podman into RDO/TripleO and the hands of its users. I have a bit of a knee-jerk reaction whenever I see someone talk about "fixing CI" when the underlying problem is in the software being tested and not the CI jobs. -- Jeremy Stanley
Hi Luke Short, On Thu, Jan 9, 2020 at 4:40 AM Luke Short <ekultails@gmail.com> wrote:
Hey folks,
Thank you for all of the feedback so far. The goal is definitely to fix this everywhere we can, no just in CI. Sorry for my poor choice of words. I will migrate this discussion over to the RDO community.
So iiuc the problem correctly podman 1.6.4 is needed to fix some race issues, the corresponding bug https://bugs.launchpad.net/tripleo/+bug/1856324 mainly referred CentOS7 jobs/users as most of the Upstream work/development around CentOS7 but considering efforts around CentOS8 i will try to put info related to both wrt RDO. With respect to CentOS8:- So plan for master is to move to CentOS8, but CentOS8 is still not completely ready, it's WIP. Current status and issues can be found with [1][2]. wrt podman version, as soon as job/users start consuming CentOS8, podman version whatever shipped with it will be available, most likely it will be podman-1.4.2-5 looking at Stream content [5], which might be updated with future updates/releases. I guess similar race issue might be hitting in Train as well, so with respect to Train, there is also plan to add CentOS8 support for Train in addition to CentOS7 as a follow up/parallel to master efforts. Now with respect to CentOS7:- Current podman version we have in RDO is 1.5.1-3 for both train and master. There was an attempt [3] in past from @Emilien Macchi to update podman to 1.6.1 in RDO but there were some issues running on CentOS7 and we didn't hear much from container Team on how to move forward, we can attempt again to see if > 1.6.1 is working which mostly depends on Container Teams plan for podman and CentOS7. In RDO we use the builds done by Container Team and last successful build on CBS is 1.6.2[4]. [1] https://lists.rdoproject.org/pipermail/dev/2020-January/009230.html [2] https://trello.com/c/fv3u22df/709-centos8-move-to-centos8 [3] https://review.rdoproject.org/r/#/c/23449/ [4] https://cbs.centos.org/koji/packageinfo?packageID=6853 [5] http://mirror.centos.org/centos/8-stream/AppStream/x86_64/os/Packages/
Sincerely, Luke Short
On Wed, Jan 8, 2020 at 5:55 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-08 15:35:03 -0700 (-0700), Alex Schultz wrote:
On Wed, Jan 8, 2020 at 3:30 PM Jeremy Stanley <fungi@yuggoth.org> wrote:
On 2020-01-08 16:49:15 -0500 (-0500), Luke Short wrote:
We have been running into a situation where an older version of Podman is used in CI that has consistent failures. It has problems deleting storage associated with a container. [...] The question/ask I have is can we ship/use a newer version of Podman in our upstream CI? Or should we continue our efforts on making a workaround? [...]
This sounds like a problem users of your software could encounter in production. If so, how does only fixing it in CI jobs help your users? It seems like time might be better spent fixing the problem for everyone.
Btw fixing CI implies fixing for everyone. In other words, how do we make it available for everyone (including CI). This is one of those ecosystem things because we (tripleo/openstack) don't necessarily ship it but we do need to use it. I'm uncertain of the centos7/podman 1.6 support and which branches are affected by this? This might be a better question for RDO.
I see, "ship/use a newer version of Podman in our upstream CI" didn't seem to necessarily imply getting a newer version of Podman into RDO/TripleO and the hands of its users. I have a bit of a knee-jerk reaction whenever I see someone talk about "fixing CI" when the underlying problem is in the software being tested and not the CI jobs. -- Jeremy Stanley
Thanks and Regards Yatin Karel
One of the things I am working on is to add CI jobs on podman project itself that builds rpms packages for all supported systems and tests them, final goal being to test them with openstack. I am not done yet but got a good progress: https://review.rdoproject.org/zuul/buildsets?project=containers%2Flibpod <https://review.rdoproject.org/zuul/buildsets?project=containers/libpod> https://github.com/containers/libpod/pull/4815 <https://github.com/containers/libpod/pull/4815> - current WIP (after merging few others) Since I started working on this I found several bugs in podman, so I think that the effort would pay off. Cheers Sorin
On 8 Jan 2020, at 22:54, Jeremy Stanley <fungi@yuggoth.org> wrote:
ewer version of Podman in our upstream CI" didn't seem to necessarily imply getting a newer version of Podman into RDO/TripleO and the hands of its users. I have a bit of a knee-jerk reaction
participants (5)
-
Alex Schultz
-
Jeremy Stanley
-
Luke Short
-
Sorin Sbarnea
-
Yatin Karel