[openstack-ansible] dropping suse support
Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi... [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi...
Hello, On 08/03/2019 16:36, Mohammed Naser wrote:
Hi everyone, [...] With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore.
Yes, I apologize for that but I have changed roles within SUSE so it became very difficult for me to maintain it myself. However, I would suggest to keep it for this cycle (as we are almost done with it) hoping that someone else will be able to take care of that during the stabilization period. I would even suggest to move the jobs to the experimental on the next cycle if nobody steps up because removing (and possibly adding back in the future) the support in the AIO and all the roles will be just too much work for anyone to pick it up. Would it be a huge problem to have it as experimental-only instead of dropping it completely? -- markos SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
On Fri, Mar 8, 2019 at 11:51 AM Markos Chandras <mchandras@suse.de> wrote:
Hello,
On 08/03/2019 16:36, Mohammed Naser wrote:
Hi everyone, [...] With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore.
Yes, I apologize for that but I have changed roles within SUSE so it became very difficult for me to maintain it myself.
First off, no need to apologize! Thank you so much for the help that you put in over the past few months in order to get it working.
However, I would suggest to keep it for this cycle (as we are almost done with it) hoping that someone else will be able to take care of that during the stabilization period.
I worry that the issue with this is that we have to maintain it for 2-3 more cycles, as any problems that it have will still have impact in our stable gates, so we're likely on the hook for maintaining it for 3 more cycles. It's also quite hard for us to drop support mid-way through a release, which is where it can get a bit hard.
I would even suggest to move the jobs to the experimental on the next cycle if nobody steps up because removing (and possibly adding back in the future) the support in the AIO and all the roles will be just too much work for anyone to pick it up.
Would it be a huge problem to have it as experimental-only instead of dropping it completely?
I think we could move the jobs to experimental. However, I don't think we've had much volunteers that wanted to step in to look into it. My thought process was to somehow make the remove one big commit within a role so all it would involve is reverting the commit that removed the support. The reason behind that is it would probably help simplify our roles rather than us trying to "workaround" something that might just eventually suffer from bitrot. I am open to all suggestions and I'd love to hear input from the community as well.
-- markos
SUSE LINUX GmbH | GF: Felix Imendörffer, Jane Smithard, Graham Norton HRB 21284 (AG Nürnberg) Maxfeldstr. 5, D-90409, Nürnberg
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com
I would be very sad to see this go, I actually run openSUSE in my home lab, though I am by no means an active user in production. Maybe this is something we can keep in tree but remove the tests for so it’s not consuming gate resources? This would be similar to the recent addition of Gentoo which has very limited, if any, gate testing at this time. If we’re still running into issues or if there are no known maintainers/users in production by B1 within the T cycle I’d be onboard with it being dropped. -- Kevin Carter IRC: Cloudnull From: Mohammed Naser Sent: Friday, March 8, 2019 10:37 AM To: OpenStack Discuss Subject: [openstack-ansible] dropping suse support Hi everyone, I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible. Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios. Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of. However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up. We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted. With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore. ..Really wish I didn't have to write this email Mohammed [1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi... [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi...
On Fri, Mar 8, 2019 at 12:34 PM Kevin Carter <kevin@cloudnull.com> wrote:
I would be very sad to see this go, I actually run openSUSE in my home lab, though I am by no means an active user in production. Maybe this is something we can keep in tree but remove the tests for so it’s not consuming gate resources? This would be similar to the recent addition of Gentoo which has very limited, if any, gate testing at this time. If we’re still running into issues or if there are no known maintainers/users in production by B1 within the T cycle I’d be onboard with it being dropped.
Given the initial response, I think I will propose changes to move the jobs to experimental. If we don't see much patches flowing in or the jobs often ran (which Zuul should provide us that information easily), we can pull the content out afterwards. I appreciate the feedback! :)
--
Kevin Carter
IRC: Cloudnull
From: Mohammed Naser Sent: Friday, March 8, 2019 10:37 AM To: OpenStack Discuss Subject: [openstack-ansible] dropping suse support
Hi everyone,
I've been trying to avoid writing this email for the longest time
ever. It's really painful to have to reach this state, but I think
we've hit a point where we can no longer maintain support for SUSE
within OpenStack Ansible.
Unfortunately, with the recent layoffs, the OSA team has taken a huge
hit in terms of contributors which means that there is far less
contributors within the project. This means that we have less
resources to go around and it forces us to focus on a more functional,
reliable and well tested set of scenarios.
Over the past few cycles, there has been effort to add SUSE support to
OpenStack Ansible, during the time that we had maintainers, it was
great and SUSE issues were being fixed promptly. In addition, due to
the larger team at the time, we found ourselves having some extra time
where we can help unbreak other gates. Jesse used to call this a
"labour of love", which I admired at the time and hoped we continue to
do as much as we can of.
However, the lack of a committed maintainer for OpenSUSE has resulted
in constantly failing jobs[1][2] (which were moved to non-voting,
wasting CI resources as no one fixed them). In addition, it's causing
several gate blocks for a few times with no one really finding the
time to clean them up.
We are resource constrained at this point and we need the resource to
go towards making the small subset of supported features functional
(i.e. CentOS/Ubuntu). We struggle with that enough, and there seems
to be no deployers that are running SUSE in real life at the moment
based on bugs submitted.
With that, I propose that we drop SUSE support this cycle. If anyone
would like to volunteer to maintain it, we can review that option, but
that would require a serious commitment as we've had maintainers step
off and it hurts the velocity of the project as no one can merge code
anymore.
..Really wish I didn't have to write this email
Mohammed
[1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi...
[2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi...
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com
On 08/03/2019 18:12, Mohammed Naser wrote:
Given the initial response, I think I will propose changes to move the jobs to experimental.
If we don't see much patches flowing in or the jobs often ran (which Zuul should provide us that information easily), we can pull the content out afterwards.
I appreciate the feedback! :)
Whilst it is sad to see the suse jobs moved to experimental, I do agree with Mohammed that under the circumstances this is the right thing to do. We need to keep our velocity up in order to address issues for end users in a timely manner, and to make best use of everyones time. Keeping the suse support code in-tree for the time being should present little if any overhead. Jon.
Hi Mohammed, Am Fr., 8. März 2019 um 17:37 Uhr schrieb Mohammed Naser <mnaser@vexxhost.com>:
ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible.
What were the previous occurences of this? I do remember that occassional IRC ping, but not a lot of activity about broken openSUSE builds. I would like to take different action here: Leap 42.3 support should indeed be moved to non-voting or removed completely, because the operating system is nearing end of life. We should be able to keep Leap 15 (15.0 currently, 15.1 when it is released) up and voting though with some initial fix up. it seems most of them are failing with: fatal: [localhost]: FAILED! => {"changed": false, "checksum": "6935ee3309d2c7b0f659e7f4ac81fa6a72569ac5", "msg": "Destination directory /var/lib/machines/opensuse-15.0-amd64/opt does not exist"} which is probably some image generation/caching logic that needs to be fixed. any idea where that is coming from and why it works for other distributions? TIA, Dirk
On Mon, Mar 11, 2019 at 9:12 AM Dirk Müller <dirk@dmllr.de> wrote:
Hi Mohammed,
Am Fr., 8. März 2019 um 17:37 Uhr schrieb Mohammed Naser <mnaser@vexxhost.com>:
ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible.
What were the previous occurences of this? I do remember that occassional IRC ping, but not a lot of activity about broken openSUSE builds.
I would like to take different action here: Leap 42.3 support should indeed be moved to non-voting or removed completely, because the operating system is nearing end of life.
We should be able to keep Leap 15 (15.0 currently, 15.1 when it is released) up and voting though with some initial fix up. it seems most of them are failing with:
fatal: [localhost]: FAILED! => {"changed": false, "checksum": "6935ee3309d2c7b0f659e7f4ac81fa6a72569ac5", "msg": "Destination directory /var/lib/machines/opensuse-15.0-amd64/opt does not exist"}
which is probably some image generation/caching logic that needs to be fixed. any idea where that is coming from and why it works for other distributions?
That's because the images don't have /opt: https://review.openstack.org/#/c/642079/ The fix doesn't seem to pass either.
TIA, Dirk
-- Mohammed Naser — vexxhost ----------------------------------------------------- D. 514-316-8872 D. 800-910-1726 ext. 200 E. mnaser@vexxhost.com W. http://vexxhost.com
On 08/03/2019 16:36, Mohammed Naser wrote:
Hi everyone,
I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible.
Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios.
Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of.
However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up.
We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted.
With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore.
..Really wish I didn't have to write this email Mohammed
[1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi... [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi...
It's now over 12 months since Mohammed posted this and the situation remains as described. It feels like time to reduce the OSA CI surface area to enable patches to merge and release contributor cycles for new work such as support for Ubuntu Focal and Centos8. Two alternative patches are proposed, one to remove support immediately [1] and another to move the current jobs to non-voting with a view to removing them entirely for the Victoria cycle [2]. [1] https://review.opendev.org/725541 [2] https://review.opendev.org/725598 Jon.
I agree, without support for it I think it's well more then time to remove it. Amy (spotz) On Tue, May 5, 2020 at 8:52 AM Jonathan Rosser <jonathan.rosser@rd.bbc.co.uk> wrote:
On 08/03/2019 16:36, Mohammed Naser wrote:
Hi everyone,
I've been trying to avoid writing this email for the longest time ever. It's really painful to have to reach this state, but I think we've hit a point where we can no longer maintain support for SUSE within OpenStack Ansible.
Unfortunately, with the recent layoffs, the OSA team has taken a huge hit in terms of contributors which means that there is far less contributors within the project. This means that we have less resources to go around and it forces us to focus on a more functional, reliable and well tested set of scenarios.
Over the past few cycles, there has been effort to add SUSE support to OpenStack Ansible, during the time that we had maintainers, it was great and SUSE issues were being fixed promptly. In addition, due to the larger team at the time, we found ourselves having some extra time where we can help unbreak other gates. Jesse used to call this a "labour of love", which I admired at the time and hoped we continue to do as much as we can of.
However, the lack of a committed maintainer for OpenSUSE has resulted in constantly failing jobs[1][2] (which were moved to non-voting, wasting CI resources as no one fixed them). In addition, it's causing several gate blocks for a few times with no one really finding the time to clean them up.
We are resource constrained at this point and we need the resource to go towards making the small subset of supported features functional (i.e. CentOS/Ubuntu). We struggle with that enough, and there seems to be no deployers that are running SUSE in real life at the moment based on bugs submitted.
With that, I propose that we drop SUSE support this cycle. If anyone would like to volunteer to maintain it, we can review that option, but that would require a serious commitment as we've had maintainers step off and it hurts the velocity of the project as no one can merge code anymore.
..Really wish I didn't have to write this email Mohammed
[1]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi... [2]: http://zuul.opendev.org/t/openstack/builds?job_name=openstack-ansible-functi...
It's now over 12 months since Mohammed posted this and the situation remains as described.
It feels like time to reduce the OSA CI surface area to enable patches to merge and release contributor cycles for new work such as support for Ubuntu Focal and Centos8.
Two alternative patches are proposed, one to remove support immediately [1] and another to move the current jobs to non-voting with a view to removing them entirely for the Victoria cycle [2].
[1] https://review.opendev.org/725541 [2] https://review.opendev.org/725598
Jon.
participants (6)
-
Amy Marrich
-
Dirk Müller
-
Jonathan Rosser
-
Kevin Carter
-
Markos Chandras
-
Mohammed Naser