On Tuesday, 11 December 2018 23:42:40 CET Thomas Goirand wrote:
Hi Luigi,
First thanks for communicating about this.
I'm afraid you probably will not like much what I'm going to have to write here. Sorry in advance if I'm not enthusiastic about this.
On 12/11/18 9:54 PM, Luigi Toscano wrote:
== Packaging
It would be nice if users could get all the plugins packages by installing the same packages as before the split, but I understand that it may be tricky, because each plugin depends on the core of sahara. This means that a binary subpackage like openstack-sahara (RDO) or sahara (Debian) can't depend on the binary packages of the plugins, or that would introduce a circular dependency.
I raised the issue on the RDO list to get some initial feedback and the solution that we are going to implement in RDO requires flipping the value of a variable to handle the bootstrap case and the normal case, please check: https://lists.rdoproject.org/pipermail/dev/2018-November/008972.html> and the rest of the thread, like: https://lists.rdoproject.org/pipermail/dev/2018-December/008977.html
Any additional idea from other packagers would be really appreciated, also because it has a direct impact on at least the puppet modules.
We aren't "packagers", we *maintain* packages, and therefore are package maintainers. And from that perspective, such decision from upstream is always disruptive, both from the package maintainer view, and for our users.
About the terminology, I'd like to point out that I've seen "packagers" used without any particular weird meaning attached in all the development communities I followed or I contributed to, even used by the people in charge of the packages, and I knew quite a lot of them. Going to the proper point:
While a plugin architecture looks sexy from the outside, it can bite hard in many ways.
In argumentation in the spec, the one and only thing which is annoying your users (according to the argumentation) is that plugins are tight to a milestone, and that it's not possible to upgrade a plug-in separately. I seriously doubt that Sahara users are going to do that, for many reasons. Here's 2 of them.
First, because if they consume packages, I don't see downstream distribution making update to packages so often. Especially if there's so many to update, and so many combination to test, it's going to be very difficult for downstream distribution to know what to do, and do regular plugin updates.
That's about people in charge of the packages, so me (partially for RDO), you, and other people.
Second, because I very much doubt that users will want to update their plugins that often. Most OpenStack users are struggling with updates of the OpenStack core already.
We didn't come out with this requirement outselves. We had various operators over the year who could not provide newer plugins to their users because they could not upgrade the whole of OpenStack and asked for this. This feature is there exactly to not upgrade the core of OpenStack. Upgrading leaves components like the plugin is a totally different story.
Another thing is experience we had with Cinder and Neutron. As you know, Cinder carries the drivers in the main repository, while Neutron decided to move to a plugin system. So we do trust the Cinder team to do a good job to maintain the plugins, and have them in sync with the release, for some Neutron plugins, it has been a complete disaster, with some plugins always lagging behind the needed API changes and so on. As for downstream distribution, it is very hard to know which plugins are worth packaging or not, and even harder to know which are well maintained upstream or not. What's your plan for addressing this issue?
By keeping a proper set of constraints in the plugins. As I mentioned, the core of Sahara is more than stable and most of the changes are in the plugins. tl;dr we don't plan API changes in the interface between the core and the plugins. And we probably don't want them. At most we may add some of them, but not remove.
As for myself, I'm really not sure I'm going to invest so much time on Sahara packaging plugin. Maybe I'll decide to package a few, but probably not all plugins, as it's very time consuming. Maybe I'll find spare cycles. It wont be important anyway, because Debian Buster will be frozen, so I may just skip that work for the Stein release, as I'll probably be busy on other stuff. So, not best timing for me as a Debian package maintainer. :/
So, maybe you could reconsider? Or is it already too late?
There is no going back.
== Preview
The work-in-progress repositories are available here:
Sahara: https://github.com/tellesnobrega/sahara/tree/split-plugins
Plugins: * Ambari: https://github.com/tellesnobrega/sahara-plugin-ambari * CDH: https://github.com/tellesnobrega/sahara-plugin-cdh * MapR: https://github.com/tellesnobrega/sahara-plugin-mapr * Spark: https://github.com/tellesnobrega/sahara-plugin-spark * Storm: https://github.com/tellesnobrega/sahara-plugin-storm * Vanilla: https://github.com/tellesnobrega/sahara-plugin-vanilla
Oh gosh... This becomes very embarrassing. I have absolutely no clue what these plugins are for, and there's absolutely no clue in the repositories. Appart from "Spark", which I vaguely heard about, the above names give zero clue on what they do. There's no hyperlink to the projects these plugins are supporting. I have no clue which of the above plugins is for Hadoop, which was historically the first target of Sahara.
That's the meaning of "testing repository"; they are not final (they will need at least one or two more rebases from the current master). The description in setup.cfg can certainly be extended, and the related content of README.rst too. This is a valuable suggestion. Ciao -- Luigi