Hi Stig, Actually, this driver was built from the start to be depending on another project's external Helm charts to exist. I've actually reached out in a review on August 22nd 2022 on an initial small stub asking if there's anything more to it: https://review.opendev.org/c/openstack/magnum/+/851076/5 The answer was no, that's as far as we've got. Hearing or seeing no updates in there, we decided to start our effort on October 20th 2022, a whole 2 months after when that patch. The next revision of that patchset came in in November 14th 2022. https://github.com/vexxhost/magnum-cluster-api/commit/8444be3a6c1b62a99a8775... I don't think we were not participating. At that point, I get removed from the CC'd on the patch in April 2023, and after a request to re-license our driver to Apache 2.0 and code which seems practically the same start to show up: https://review.opendev.org/c/openstack/magnum/+/851076/19/magnum/drivers/clu... I think it is massively unfair to first say we didn't participate, we actually tried and lead the whole thing. I also think it is not great for open source to assume that everyone will be Hashicorp or an Elastic. If we're going to be operating under that general assumption, a lot of this stuff is going to fall apart. If "bad intentions" are assumed out of the box, you're seeming to mention that the only way to do open source is under "open governance" Once again, I expressed MANY times that we will gladly contribute the driver if we see folks contributing to it, however we've yet to see that, so I don't see why we should slow everything down to satisfy a group that doesn't contribute anything to the driver. At this point, the contributed driver is "behind" the "actual" development happening out of an open governance: https://github.com/stackhpc/magnum-capi-helm We can see some folks from Catalsyt Cloud which have decided to work with you to push the driver contributing to that repository. There is also dependencies on image builds which are pointing to your servers. There is dependencies on Helm charts that are living on your GitHub pages, with no code being uploaded since the repository was created for over 6 months: https://opendev.org/openstack/magnum-capi-helm-charts So to me, it feels like there is efforts happening somewhere else, and now they're just being pushed in now. If you truly believed in what you were saying, this could have easily been an OpenDev repo, an extra project under the Magnum governance, and all the Helm chart development would have lived there. Instead, the work is ALL done elsewhere, and now it's being code dumped into Magnum. I hope you see my concerns with all this. This is why a driver that's mostly been developed out of tree, with artifacts living out of tree (arguably, the Helm charts are the biggest most critical dependency and that's not being developed in governance) being merged is concerning. Does this mean it's okay to setup a job that just code dumps our code straight into Magnum while we continue to work on it elsewhere and that's fine? Mohammed ________________________________ From: Stig Telfer <stig.openstack@telfer.org> Sent: February 20, 2024 5:43 AM To: Mohammed Naser <mnaser@vexxhost.com> Cc: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [magnum] Dropping default Cluster API driver You don't often get email from stig.openstack@telfer.org. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hello Mohammed - Thank you for clarifying your point about merging the driver. I agree the ability to select drivers is important, not least because any new driver must coexist with the Heat driver for continuity on existing deployments. On your point about the four opens, I think this is key. The governance clearly matters, when we consider Elastic and Hashicorp (as mentioned previously on this thread). You could well say that Vexxhost is different, but bear in mind that Hashicorp and Elastic were also champions of open source licensing - until they weren't. From my point of view it is unfortunate that Vexxhost did not contribute to the upstream development of the Cluster API Helm driver which predated Vexxhost's internal development. I think I understand your rationale for that choice - developing in the four opens can be fraught for something on which business depends. However it is a strategy that brings its own risks, which is more general than Magnum and sometimes results in outcomes of this nature. In a previous role I have also had direct experience of that. Best wishes, Stig On 17 Feb 2024, at 13:21, Mohammed Naser <mnaser@vexxhost.com> wrote: Hi Stig: I'm surprised because there was no consensus on that in the vPTG, because we went back and forth for quite a long time in that conversation. Also, I'm not sure what the four opens have to do with this. The open core comments have been thrown around a few times and our work is all open, perhaps it's the fact that it doesn't live under the governance which is the big issue here? I've stated many times that we'd gladly upstream it if we saw any contributions at all, but we haven't .. so we kept on chugging along on our own pace. Also, I'm not saying to not merge it. I'm saying that Magnum should out of the box allow the user to pick what driver they want. The deployment tools will decide what to support. Thanks Mohammed Get Outlook for iOS<https://aka.ms/o0ukef> ________________________________ From: Stig Telfer <stig.openstack@telfer.org> Sent: Saturday, February 17, 2024 4:43:47 AM To: Mohammed Naser <mnaser@vexxhost.com> Cc: Dmitriy Rabotyagov <noonedeadpunk@gmail.com>; OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [magnum] Dropping default Cluster API driver You don't often get email from stig.openstack@telfer.org. Learn why this is important<https://aka.ms/LearnAboutSenderIdentification> Hello Mohammed - I am surprised at your reaction to this. Broadly speaking this driver is a subject that has been undergoing the open design process since the Yoga PTG, and i do agree with Dmitriy that the consensus in the last PTG was that there is benefit to an in-tree driver. Development does appear to be following a course. The October 2023 PTG etherpad also begins with notes that the CAPI Helm driver cannot be merged until Magnum is refactored to accommodate it alongside the Vexxhost driver without conflict. There has been a delay of at least two cycles to support that. In short I don't think there are negative user impacts for Vexxhost's driver by this driver's introduction. The Vexxhost team has benefited from a rapid pace of development by doing so outside of OpenStack's governance and the four opens. Is it fair to obstruct the delivery of a driver that has attempted to follow those guidelines? Best wishes, Stig On 16 Feb 2024, at 20:45, Mohammed Naser <mnaser@vexxhost.com> wrote: Hi Dmitriy, I am really glad you're taking the time to respond – any feedback is greatly appreciated. With regards to the service being "functional" out of the box. The way that I see this, is that this is a decision that needs to be taken by the operator: what is the backend that I will be using? I see this in the same way that "Cinder" out of the box has no default: you need a storage backend. While we (I think?) mostly test with LVM in the CI (and some Ceph CI and what not). Deployment tools can be a bit "nuanced" in that they may support one backend over another out of the box, but for something like Cinder, you need to decide what you're using. With regards to those "drivers" not being usable or not being able to have "influence". I can't speak on any of the other ones, but on our side, I'm hoping we have enough "good karma" with the community knowing that we're welcome at working together with folks. I can't speak on behalf of others but we've had contributions from countless organizations that have added features such as Flatcar, support for HTTP proxies, and many other changes that we've merged with no issues from those organizations. I don't entirely agree on the Cinder aspect, since Cinder third party CI mostly exists because OpenDev doesn't want to own a rack with 40 different types of storage systems.. but instead the 'built-in' CI works for the open source software backends (say, Ceph or LVM) and then third party is needed for those needing physical access to the machines. I think a CI that tests drivers would be productive if there's a "no built-in approach". In this case, I do understand the concern that it seems to feel like a single organization driven project -- however, we've had contributors from all over – https://github.com/vexxhost/magnum-cluster-api/graphs/contributors – and the way I would see this is .. how is this different from any other OpenStack dependency if it was an "add-on". Silly example, `confluent-kafka` is a dependency of oslo.messaging if you're using Kafka as a backend. There's probably many other examples of other libraries that exist as OpenStack dependencies that might not even be single organization driven, but single "individual" driven... I can't speak on the rest, but we've been asked to license the code as Apache 2.0 – which it has been .. just like any other OpenStack dependency. Magnum is really good at being a standard API that implements many backends, and the user should choose what backend they want. In the same way that Cinder is same way allowing multiple backends, and letting the user pick what they want. Once again though, I really appreciate your thoughts on this and getting the conversation going, Dmitriy. Thanks, Mohammed ________________________________ From: Dmitriy Rabotyagov <noonedeadpunk@gmail.com> Sent: February 16, 2024 3:24 PM Cc: OpenStack Discuss <openstack-discuss@lists.openstack.org> Subject: Re: [magnum] Dropping default Cluster API driver Hey, I'm totally not to decide here, but just want to repeat my personal subjective opinion on the topic. From what I do recall from the last PTG, which I was part of, more opinions were inclined towards having some default driver. Otherwise, service is not functional on it's own and must rely heavily on individual organisations to keep support for their drivers without any way of influencing them. In turn, organisations are able to dictate how project should be developed, what changes they are able or not able to do. I'd say, this suggestion is similar to drop native driver from Octavia or Cinder and just assume that individual organisations will keep support of their drivers. With that, these service doing just fine with having some default drivers, and some third-party. Also adding third party CI to the service is not huge issue, to ensure that your driver integrates nicely. With that said, OpenStack-Ansible right now is merging support and CI testing of Vexxhost driver. I also personally highly likely will be using it in future magnum deployments. But I truly believe, that service without any driver is doomed to failure sooner then later. If we look at literally anything - automation tool, programming language, service - they all have some core libraries, and can be used, in a way, "out of the box". And here suggestion is to make service and all it's users to put faith in an organisation providing support for the driver, which may be quite tough sell for some highly regulated environments. In case Magnum is going the road of not having any "out-of-the-box" implementation, i actually wonder if it should stay under OpenStack namespace rather then X, for instance, since it's not self-contained and heavily depends on third party, which might be licensed in a completely different way, as well as change their licensing with time which also put users at tough spot. But dunno, as I said, this is my very subjective view, and I really can be wrong in this. Also not trying to push any party to any conclusion. So treat it as - one operator voice out of many. On Fri, Feb 16, 2024, 21:02 Mohammed Naser <mnaser@vexxhost.com<mailto:mnaser@vexxhost.com>> wrote: Hi everyone, It seems that based on a recent post that I saw on the Twitter-sphere[1], it seems that the Magnum team has decided to move forward with setting a default driver for the Cluster API. I don't agree with this. Unfortunately, we don't have much recordings of our PTG conversations, and since the notes on the PTG are pretty light[2], I don't see any details of that. I don't see why we should be merging a native Cluster API driver and not allowing the operators and deployment tools decide which one they want to support. Personally, I think that the Helm driver should be built in a separate repository and then installed by the user as a Python package once they understand what are the backends and their options, as opposed to shipping one out of the box with far less features. We already have successful users which are using our implementation in production that have also posted blogs and shared their experience, we'd be doing them a disservice by now announcing a 'native' built-in driver. Can the decision be changed so that there is no built-in driver, and you must decide to install one yourself? Also, I would be really happy to hear feedback from users of the driver for their comments here. Thanks, Mohmamed [1]: https://www.stackhpc.com/magnum-clusterapi.html [2]: https://etherpad.opendev.org/p/r.5b94a5a9dbf4e3bf8202b5e57cb93770