RE: Re: New Openstack Deployment questions - how will we respond to this change?
The purpose of my Openstack Deployment email was not only to rant about the decision to kill Centos 8. I want to make sure that we don't blindly go down the easy route of pretending that Centos Stream is an acceptable replacement for Centos. If we do this, we would do a disservice to the community and to everyone who depends on it. The reality is that, at the end of 2021, Centos will no longer exist as a viable production operating system, and everyone who was using it will have to adjust. For companies that already have Centos mirror repositories setup and people managing them, the increased load imposed by Stream will be incremental, but I suspect that the vast majority of companies do not have that infrastructure already setup, and we will need to carefully consider whether our needs are better met by going forward with Stream or switching to another OS. If the OpenStack community decides to continue building on Stream, we should make it crystal clear to operators and users that Stream is not a production-ready OS and that our Stream OpenStack implementation is suitable for testing and development use only, unless they devote substantial resources to mirroring and testing Stream to insulate production clusters from the instability that it will introduce. If we do decide to continue building on Stream, how will we respond to the constant stream of bug reports that will result when Stream changes cause OpenStack components to fail? Many component teams are already understaffed. Will we have the bandwidth to constantly chase changes? Should we replace our Centos builds with RHEL, or with Rocky? Does the community have (or can we find) the resources to do the work of maintaining stable Stream mirrors and only building OpenStack on our stable versions of Stream? Or would it be better to drop Centos support and focus our efforts on operating systems that have not implemented unilateral changes that harm the community? -----Original Message----- From: Braden, Albert Sent: Friday, December 11, 2020 9:10 AM To: openstack-discuss@lists.openstack.org Subject: RE: [EXTERNAL] Re: New Openstack Deployment questions Centos Stream is fine for those who were using Centos for testing or development. It's not at all suitable for production, because rolling release doesn't provide the stability that production clusters need. Switching to Centos Stream would require significant resources to be expended to setup local mirrors and then perform exhaustive testing before each upgrade. The old Centos did this work for us; Centos was built on RHEL source that had already been tested by paying customers, and bugs fixed with the urgency that paying customers require. Adding an upstream build (Stream) to the existing downstream (Centos 8.x) was fine, but I'm disappointed by the decision to kill Centos 8. I don't want to wax eloquent about how we were betrayed; suffice it to say that even for a free operating system, suddenly changing the EOL from 2029 to 2021 is unprecedented, and places significant burdens on companies that are using Centos in production. I can understand why IBM/RH made this decision, but there's no denying that it puts production Centos users in a difficult position. I hope that Rocky Linux [1], under Gregory Kurtzer (founder of the Centos project) will turn out to be a useful alternative. {1} https://github.com/rocky-linux/rocky -----Original Message----- From: Luigi Toscano <ltoscano@redhat.com> Sent: Thursday, December 10, 2020 3:51 PM To: Thomas Wakefield <dwakefi2@gmu.edu>; openstack-discuss@lists.openstack.org Cc: Satish Patel <satish.txt@gmail.com> Subject: [EXTERNAL] Re: New Openstack Deployment questions CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance. On Thursday, 10 December 2020 15:27:40 CET Satish Patel wrote:
I just built a new openstack using openstack-ansible on CentOS 8.2 last month before news broke out. I have no choice so i am going to stick with CentOS.
What is the future of RDO and EPEL repo if centOS going away. ?
Continue as before on CentOS Stream. -- Luigi I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it. E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On Tue, Dec 15, 2020 at 7:46 AM Braden, Albert <C-Albert.Braden@charter.com> wrote:
The purpose of my Openstack Deployment email was not only to rant about the decision to kill Centos 8. I want to make sure that we don't blindly go down the easy route of pretending that Centos Stream is an acceptable replacement for Centos. If we do this, we would do a disservice to the community and to everyone who depends on it. The reality is that, at the end of 2021, Centos will no longer exist as a viable production operating system, and everyone who was using it will have to adjust. For companies that already have Centos mirror repositories setup and people managing them, the increased load imposed by Stream will be incremental, but I suspect that the vast majority of companies do not have that infrastructure already setup, and we will need to carefully consider whether our needs are better met by going forward with Stream or switching to another OS.
From an OpenStack development perspective, centos8 stream is an acceptable centos replacement because it finally allows us to get a
head of "stable" versions and get the issues resolved early and get fixes committed upstream faster. It may not be an acceptable replacement for operators, but that's up to the operator to determine. An operator can delay updates from stream and implement their own rolling updates as necessary via CI/CD (via repo mirroring) which previously was not the case when CentOS dropped exposing specific minor point releases. Since the inception of 8, there hasn't been an 8.1 or 8.2 you could pin to. You got rolling updates whenever 8.2 hit, you got it without being able to go back.
If the OpenStack community decides to continue building on Stream, we should make it crystal clear to operators and users that Stream is not a production-ready OS and that our Stream OpenStack implementation is suitable for testing and development use only, unless they devote substantial resources to mirroring and testing Stream to insulate production clusters from the instability that it will introduce.
The determination of a 'production-ready' OS is likely not something that OpenStack should be expressing. IMHO, that decision is an end user decision based on their comfort for the risk involved. From an RDO/TirpleO perspective, we will be aligning on CentOS Stream going forward so we will be addressing issues as they come up. We will likely need to improve our documentation around supported module versions, etc.
If we do decide to continue building on Stream, how will we respond to the constant stream of bug reports that will result when Stream changes cause OpenStack components to fail? Many component teams are already understaffed. Will we have the bandwidth to constantly chase changes? Should we replace our Centos builds with RHEL, or with Rocky? Does the community have (or can we find) the resources to do the work of maintaining stable Stream mirrors and only building OpenStack on our stable versions of Stream? Or would it be better to drop Centos support and focus our efforts on operating systems that have not implemented unilateral changes that harm the community?
TBH I don't think things change. We've always had to respond to breaking versions even with CentOS. We just hit a fair number of issues with 8.3 being released last week. I think this actually reduces the impact because we're likely to get individual failures that can be identified/addressed instead of large sweeping changes that we get when there's an 8.2 -> 8.3 transition.
-----Original Message----- From: Braden, Albert Sent: Friday, December 11, 2020 9:10 AM To: openstack-discuss@lists.openstack.org Subject: RE: [EXTERNAL] Re: New Openstack Deployment questions
Centos Stream is fine for those who were using Centos for testing or development. It's not at all suitable for production, because rolling release doesn't provide the stability that production clusters need. Switching to Centos Stream would require significant resources to be expended to setup local mirrors and then perform exhaustive testing before each upgrade. The old Centos did this work for us; Centos was built on RHEL source that had already been tested by paying customers, and bugs fixed with the urgency that paying customers require.
Adding an upstream build (Stream) to the existing downstream (Centos 8.x) was fine, but I'm disappointed by the decision to kill Centos 8. I don't want to wax eloquent about how we were betrayed; suffice it to say that even for a free operating system, suddenly changing the EOL from 2029 to 2021 is unprecedented, and places significant burdens on companies that are using Centos in production. I can understand why IBM/RH made this decision, but there's no denying that it puts production Centos users in a difficult position.
I hope that Rocky Linux [1], under Gregory Kurtzer (founder of the Centos project) will turn out to be a useful alternative.
{1} https://github.com/rocky-linux/rocky
-----Original Message----- From: Luigi Toscano <ltoscano@redhat.com> Sent: Thursday, December 10, 2020 3:51 PM To: Thomas Wakefield <dwakefi2@gmu.edu>; openstack-discuss@lists.openstack.org Cc: Satish Patel <satish.txt@gmail.com> Subject: [EXTERNAL] Re: New Openstack Deployment questions
CAUTION: The e-mail below is from an external source. Please exercise caution before opening attachments, clicking links, or following guidance.
On Thursday, 10 December 2020 15:27:40 CET Satish Patel wrote:
I just built a new openstack using openstack-ansible on CentOS 8.2 last month before news broke out. I have no choice so i am going to stick with CentOS.
What is the future of RDO and EPEL repo if centOS going away. ?
Continue as before on CentOS Stream.
-- Luigi
I apologize for the nonsense below. So far I have not been able to stop it from being attached to my external emails. I'm working on it.
E-MAIL CONFIDENTIALITY NOTICE: The contents of this e-mail message and any attachments are intended solely for the addressee(s) and may contain confidential and/or legally privileged information. If you are not the intended recipient of this message or if this message has been addressed to you in error, please immediately alert the sender by reply e-mail and then delete this message and any attachments. If you are not the intended recipient, you are notified that any use, dissemination, distribution, copying, or storage of this message or any attachment is strictly prohibited.
On 2020-12-15 14:40:01 +0000 (+0000), Braden, Albert wrote: [...]
If the OpenStack community decides to continue building on Stream, we should make it crystal clear to operators and users that Stream is not a production-ready OS and that our Stream OpenStack implementation is suitable for testing and development use only, unless they devote substantial resources to mirroring and testing Stream to insulate production clusters from the instability that it will introduce.
Up until https://review.openstack.org/638045 merged last year, we used to put it plainly in our testing specs that: "The following free operating systems are representative of platforms regularly used to deploy OpenStack on: [...] Latest CentOS Major [...] The CentOS distribution is derived from the sources of Red Hat Enterprise Linux (RHEL). In reality, RHEL is more popular than CentOS but we can't use this platform on upstream gates, so we rely on CentOS." In essence, we've always been targeting RHEL and using CentOS as a stand-in substitute. For that purpose, CentOS 8 Stream ought to suffice for continued testing of our future releases to make sure they remain compatible with RHEL 8. In fact, it may actually be superior for that purpose, as it allows us to test what's going to appear in impending minor and point releases of RHEL 8 rather than testing a laggy copy of what's already been added in RHEL.
Should we replace our Centos builds with RHEL, or with Rocky?
RHEL is still not a possibility for our CI from a licensing perspective, from what I understand. Rocky Linux might be a possibility, sure, once it's more than just a readme file and vapor.
Does the community have (or can we find) the resources to do the work of maintaining stable Stream mirrors and only building OpenStack on our stable versions of Stream?
We already do: <URL: https://opendev.org/opendev/system-config/src/commit/d2d06e690b2b8288d0b3e79... >
Or would it be better to drop Centos support and focus our efforts on operating systems that have not implemented unilateral changes that harm the community? [...]
The TripleO project is by far the largest user of our CI infrastructure (in aggregate node-hours), and they only work on RHEL/RDO or close derivatives like CentOS, so I expect at least they'll see value in continuing to have something RHEL-like to test changes against. -- Jeremy Stanley
participants (3)
-
Alex Schultz
-
Braden, Albert
-
Jeremy Stanley