[sahara][ops][tripleo][openstack-ansible][puppet-openstack][kolla][i18n] Upcoming split of the sahara repository[sahara][ops][tripleo][openstack-ansible][puppet-openstack][kolla][i18n] Upcoming split of the sahara repository
Hi all, we (sahara developers) are working on splitting the content of sahara.git, moving the code of each plugin to its own respository. Each plugin is used for a different Hadoop/Big Data provider. The change is better described in the following spec, which didn't happen in Rocky: http://specs.openstack.org/openstack/sahara-specs/specs/rocky/plugins-outsid... The change is going to impact other projects, but hopefully not the users. I will try to summarize the impact on the various consumers of sahara. == Deployment tools - the devstack plugin in the test repository (see below) was modified to support the installation of all the available plugins previously available in- tree, so devstack users should not notice any change. - puppet-sahara will require changes to install the additional repositories, and we would require and welcome any help; - I hope that the puppet-sahara changes would be enough to make TripleO work, but we would need some help on this. - openstack-ansibe-os_sahara will require changes as well, but we have a bit more expertise on Ansible and we can help with the changes. - kolla will require some changes too, depending on te (apologize if I forgot some project) == Packaging It would be nice if users could get all the plugins packages by installing the same packages as before the split, but I understand that it may be tricky, because each plugin depends on the core of sahara. This means that a binary subpackage like openstack-sahara (RDO) or sahara (Debian) can't depend on the binary packages of the plugins, or that would introduce a circular dependency. I raised the issue on the RDO list to get some initial feedback and the solution that we are going to implement in RDO requires flipping the value of a variable to handle the bootstrap case and the normal case, please check: https://lists.rdoproject.org/pipermail/dev/2018-November/008972.html and the rest of the thread, like: https://lists.rdoproject.org/pipermail/dev/2018-December/008977.html Any additional idea from other packagers would be really appreciated, also because it has a direct impact on at least the puppet modules. == Preview The work-in-progress repositories are available here: Sahara: https://github.com/tellesnobrega/sahara/tree/split-plugins Plugins: * Ambari: https://github.com/tellesnobrega/sahara-plugin-ambari * CDH: https://github.com/tellesnobrega/sahara-plugin-cdh * MapR: https://github.com/tellesnobrega/sahara-plugin-mapr * Spark: https://github.com/tellesnobrega/sahara-plugin-spark * Storm: https://github.com/tellesnobrega/sahara-plugin-storm * Vanilla: https://github.com/tellesnobrega/sahara-plugin-vanilla Any comment is more than welcome! Ciao -- Luigi
On Tue, Dec 11, 2018 at 3:57 PM Luigi Toscano <ltoscano@redhat.com> wrote:
Hi all,
we (sahara developers) are working on splitting the content of sahara.git, moving the code of each plugin to its own respository. Each plugin is used for a different Hadoop/Big Data provider. The change is better described in the following spec, which didn't happen in Rocky:
http://specs.openstack.org/openstack/sahara-specs/specs/rocky/plugins-outsid...
this sounds really cool, thanks for the update Luigi. peace o/
Hi Luigi, First thanks for communicating about this. I'm afraid you probably will not like much what I'm going to have to write here. Sorry in advance if I'm not enthusiastic about this. On 12/11/18 9:54 PM, Luigi Toscano wrote:
== Packaging
It would be nice if users could get all the plugins packages by installing the same packages as before the split, but I understand that it may be tricky, because each plugin depends on the core of sahara. This means that a binary subpackage like openstack-sahara (RDO) or sahara (Debian) can't depend on the binary packages of the plugins, or that would introduce a circular dependency.
I raised the issue on the RDO list to get some initial feedback and the solution that we are going to implement in RDO requires flipping the value of a variable to handle the bootstrap case and the normal case, please check: https://lists.rdoproject.org/pipermail/dev/2018-November/008972.html and the rest of the thread, like: https://lists.rdoproject.org/pipermail/dev/2018-December/008977.html
Any additional idea from other packagers would be really appreciated, also because it has a direct impact on at least the puppet modules.
We aren't "packagers", we *maintain* packages, and therefore are package maintainers. And from that perspective, such decision from upstream is always disruptive, both from the package maintainer view, and for our users. While a plugin architecture looks sexy from the outside, it can bite hard in many ways. In argumentation in the spec, the one and only thing which is annoying your users (according to the argumentation) is that plugins are tight to a milestone, and that it's not possible to upgrade a plug-in separately. I seriously doubt that Sahara users are going to do that, for many reasons. Here's 2 of them. First, because if they consume packages, I don't see downstream distribution making update to packages so often. Especially if there's so many to update, and so many combination to test, it's going to be very difficult for downstream distribution to know what to do, and do regular plugin updates. Second, because I very much doubt that users will want to update their plugins that often. Most OpenStack users are struggling with updates of the OpenStack core already. Another thing is experience we had with Cinder and Neutron. As you know, Cinder carries the drivers in the main repository, while Neutron decided to move to a plugin system. So we do trust the Cinder team to do a good job to maintain the plugins, and have them in sync with the release, for some Neutron plugins, it has been a complete disaster, with some plugins always lagging behind the needed API changes and so on. As for downstream distribution, it is very hard to know which plugins are worth packaging or not, and even harder to know which are well maintained upstream or not. What's your plan for addressing this issue? As for myself, I'm really not sure I'm going to invest so much time on Sahara packaging plugin. Maybe I'll decide to package a few, but probably not all plugins, as it's very time consuming. Maybe I'll find spare cycles. It wont be important anyway, because Debian Buster will be frozen, so I may just skip that work for the Stein release, as I'll probably be busy on other stuff. So, not best timing for me as a Debian package maintainer. :/ So, maybe you could reconsider? Or is it already too late?
== Preview
The work-in-progress repositories are available here:
Sahara: https://github.com/tellesnobrega/sahara/tree/split-plugins
Plugins: * Ambari: https://github.com/tellesnobrega/sahara-plugin-ambari * CDH: https://github.com/tellesnobrega/sahara-plugin-cdh * MapR: https://github.com/tellesnobrega/sahara-plugin-mapr * Spark: https://github.com/tellesnobrega/sahara-plugin-spark * Storm: https://github.com/tellesnobrega/sahara-plugin-storm * Vanilla: https://github.com/tellesnobrega/sahara-plugin-vanilla
Oh gosh... This becomes very embarrassing. I have absolutely no clue what these plugins are for, and there's absolutely no clue in the repositories. Appart from "Spark", which I vaguely heard about, the above names give zero clue on what they do. There's no hyperlink to the projects these plugins are supporting. I have no clue which of the above plugins is for Hadoop, which was historically the first target of Sahara. Please, please please please, for each plugin, provide: - A sort description of 80 chars, in one line - A long description of *AT LEAST* 3 lines of 80 chars That's IMO the minimum you own to the downstream package maintainers that are doing the work for your project for free (that's really my case, I never used Sahara, and I have zero professional insentive to package it, I really do it as a courtesy for Debian and OpenStack users). Thanks for working on Sahara, Cheers, Thomas Goirand (zigo) P.S: I've disregarded the added amount of work that all of this change imposes to all of us, as I thought you knew already about it, but this is not a good news to know 6 new packages will be needed... :(
On Tuesday, 11 December 2018 23:42:40 CET Thomas Goirand wrote:
Hi Luigi,
First thanks for communicating about this.
I'm afraid you probably will not like much what I'm going to have to write here. Sorry in advance if I'm not enthusiastic about this.
On 12/11/18 9:54 PM, Luigi Toscano wrote:
== Packaging
It would be nice if users could get all the plugins packages by installing the same packages as before the split, but I understand that it may be tricky, because each plugin depends on the core of sahara. This means that a binary subpackage like openstack-sahara (RDO) or sahara (Debian) can't depend on the binary packages of the plugins, or that would introduce a circular dependency.
I raised the issue on the RDO list to get some initial feedback and the solution that we are going to implement in RDO requires flipping the value of a variable to handle the bootstrap case and the normal case, please check: https://lists.rdoproject.org/pipermail/dev/2018-November/008972.html> and the rest of the thread, like: https://lists.rdoproject.org/pipermail/dev/2018-December/008977.html
Any additional idea from other packagers would be really appreciated, also because it has a direct impact on at least the puppet modules.
We aren't "packagers", we *maintain* packages, and therefore are package maintainers. And from that perspective, such decision from upstream is always disruptive, both from the package maintainer view, and for our users.
About the terminology, I'd like to point out that I've seen "packagers" used without any particular weird meaning attached in all the development communities I followed or I contributed to, even used by the people in charge of the packages, and I knew quite a lot of them. Going to the proper point:
While a plugin architecture looks sexy from the outside, it can bite hard in many ways.
In argumentation in the spec, the one and only thing which is annoying your users (according to the argumentation) is that plugins are tight to a milestone, and that it's not possible to upgrade a plug-in separately. I seriously doubt that Sahara users are going to do that, for many reasons. Here's 2 of them.
First, because if they consume packages, I don't see downstream distribution making update to packages so often. Especially if there's so many to update, and so many combination to test, it's going to be very difficult for downstream distribution to know what to do, and do regular plugin updates.
That's about people in charge of the packages, so me (partially for RDO), you, and other people.
Second, because I very much doubt that users will want to update their plugins that often. Most OpenStack users are struggling with updates of the OpenStack core already.
We didn't come out with this requirement outselves. We had various operators over the year who could not provide newer plugins to their users because they could not upgrade the whole of OpenStack and asked for this. This feature is there exactly to not upgrade the core of OpenStack. Upgrading leaves components like the plugin is a totally different story.
Another thing is experience we had with Cinder and Neutron. As you know, Cinder carries the drivers in the main repository, while Neutron decided to move to a plugin system. So we do trust the Cinder team to do a good job to maintain the plugins, and have them in sync with the release, for some Neutron plugins, it has been a complete disaster, with some plugins always lagging behind the needed API changes and so on. As for downstream distribution, it is very hard to know which plugins are worth packaging or not, and even harder to know which are well maintained upstream or not. What's your plan for addressing this issue?
By keeping a proper set of constraints in the plugins. As I mentioned, the core of Sahara is more than stable and most of the changes are in the plugins. tl;dr we don't plan API changes in the interface between the core and the plugins. And we probably don't want them. At most we may add some of them, but not remove.
As for myself, I'm really not sure I'm going to invest so much time on Sahara packaging plugin. Maybe I'll decide to package a few, but probably not all plugins, as it's very time consuming. Maybe I'll find spare cycles. It wont be important anyway, because Debian Buster will be frozen, so I may just skip that work for the Stein release, as I'll probably be busy on other stuff. So, not best timing for me as a Debian package maintainer. :/
So, maybe you could reconsider? Or is it already too late?
There is no going back.
== Preview
The work-in-progress repositories are available here:
Sahara: https://github.com/tellesnobrega/sahara/tree/split-plugins
Plugins: * Ambari: https://github.com/tellesnobrega/sahara-plugin-ambari * CDH: https://github.com/tellesnobrega/sahara-plugin-cdh * MapR: https://github.com/tellesnobrega/sahara-plugin-mapr * Spark: https://github.com/tellesnobrega/sahara-plugin-spark * Storm: https://github.com/tellesnobrega/sahara-plugin-storm * Vanilla: https://github.com/tellesnobrega/sahara-plugin-vanilla
Oh gosh... This becomes very embarrassing. I have absolutely no clue what these plugins are for, and there's absolutely no clue in the repositories. Appart from "Spark", which I vaguely heard about, the above names give zero clue on what they do. There's no hyperlink to the projects these plugins are supporting. I have no clue which of the above plugins is for Hadoop, which was historically the first target of Sahara.
That's the meaning of "testing repository"; they are not final (they will need at least one or two more rebases from the current master). The description in setup.cfg can certainly be extended, and the related content of README.rst too. This is a valuable suggestion. Ciao -- Luigi
participants (3)
-
Luigi Toscano
-
Michael McCune
-
Thomas Goirand