Open Stack

Preventing plugin developers from implementing their own installer is
a pro, not a con, you've already listed one reason in cons against
install scripts inside plugin tarball: if we centralize plugin
installation and management logic in fuel, we can change it once for
all plugins and don't have to worry about old plugins using an
obsolete installer.

I think priorities here should be 1) easy of plugin development; and
2) ease of use. Pluggable architecture won't do us much good if we end
up being the only ones being able to use it efficiently. Adding a
little more complexity to fuelclient to allow moving a lot of fuel
complexity from core to plugins is a good tradeoff.


On Thu, Oct 23, 2014 at 8:32 AM, Evgeniy L <eli at mirantis.com> wrote:
> Hi Mike,
>
> I would like to add a bit more details about current implementation and how
> it can be done.
>
> Implement installation as a scripts inside of tar ball:
> Cons:
> * install script is really simple right now, but it will be much more
> complicated
> ** it requires to implement logic where we can ask user for login/password
> ** use some config, where we will be able to get endpoints, like where is
> keystone, nailgun
> ** validate that it's possible to install plugin on the current version of
> master
> ** handle error cases (to make installation process more atomic)
> * it will be impossible to deprecate the installation logic/method, because
> it's on the plugin's side
>   and you cannot change a plugin which user downloaded some times ago, when
> we get
>   plugin manager, we probably would like user to use plugin manager, instead
> of some scripts
> * plugin installation process is not so simple as it could be (untar, cd
> plugin, ./install)
>
> Pros:
> * plugin developer can change installation scripts (I'm not sure if it's a
> pros)
>
> Add installation to fuel client:
> Cons:
> * requires changes in fuel client, which are not good for fuel client by
> design (fuel client
>   should be able to work remotely from user's machine), current
> implementation requires
>   local operations on files, it will be changed in the future releases, so
> fuel-client will
>   be able to do it via api, also we can determine if it's not master node by
> /etc/fuel/version.yaml
>   and show the user a message which says that in the current version it's
> not possible
>   to install the plugin remotely
> * plugin developer won't be able to change installation process (I'm not
> sure if it's a cons)
>
> Pros:
> * it's easier for user to install the plugin `fuel --install-plugin
> plugin_name-1.0.1.fpb'
> * all of the authentication logic already implemented in fuel client
> * fuel client uses config with endpoints which is generated by puppet
> * it will be easier to deprecate previous installation approach, we can just
> install new
>   fuel client on the master which uses api
>
> Personally I like the second approach, and I think we should try to
> implement it,
> when we get time.
>
> Thanks,
>
> On Thu, Oct 23, 2014 at 3:02 PM, Mike Scherbakov <mscherbakov at mirantis.com>
> wrote:
>>>
>>> I feel like we should not require user to unpack the plugin before
>>> installing it. Moreover, we may chose to distribute plugins in our own
>>> format, which we may potentially change later. E.g. "lbaas-v2.0.fp". I'd
>>> rather stick with two actions:
>>>
>>> Assembly (externally): fpb --build <name>
>>>
>>> Installation (on master node): fuel --install-plugin <name>
>>>
>>>  I like the idea of putting plugin installation functionality in fuel
>>> client, which is installed
>>> on master node.
>>> But in the current version plugin installation requires files operations
>>> on the master,
>>> as result we can have problems if user's fuel-client is installed on
>>> another env.
>>
>>
>> I suggest to keep it simple for now as we have the issue mentioned by
>> Evgeny: fuel client is supposed to work from other nodes, and we will need
>> additional verification code in there. Also, to make it smooth, we will have
>> to end up with a few more checks - like what if tarball is broken, what if
>> we can't find install script in it, etc.
>> I'd suggest to run it simple for 6.0, and then we will see how it's being
>> used and what other limitations / issues we have around plugin installation
>> and usage. We can consider to make this functionality as part of fuel client
>> a bit later.
>>
>> Thanks,
>>
>> On Tue, Oct 21, 2014 at 6:57 PM, Vitaly Kramskikh
>> <vkramskikh at mirantis.com> wrote:
>>>
>>> Hi,
>>>
>>> As for a separate section for plugins, I think we should not force it and
>>> leave this decision to a plugin developer, so he can create just a single
>>> checkbox or a section of the settings tab or a separate tab depending on
>>> plugin functionality. Plugins should be able to modify arbitrary release
>>> fields. For example, if Ceph was a plugin, it should be able to extend
>>> wizard config to add new options to Storage pane. If vCenter was a plugin,
>>> it should be able to set maximum amount of Compute nodes to 0.
>>>
>>> 2014-10-20 21:21 GMT+07:00 Evgeniy L <eli at mirantis.com>:
>>>>
>>>> Hi guys,
>>>>
>>>> Romans' questions:
>>>>
>>>> >> I feel like we should not require user to unpack the plugin before
>>>> >> installing it.
>>>> >> Moreover, we may chose to distribute plugins in our own format, which
>>>> >> we
>>>> >> may potentially change later. E.g. "lbaas-v2.0.fp".
>>>>
>>>> I like the idea of putting plugin installation functionality in fuel
>>>> client, which is installed
>>>> on master node.
>>>> But in the current version plugin installation requires files operations
>>>> on the master,
>>>> as result we can have problems if user's fuel-client is installed on
>>>> another env.
>>>> What we can do is to try to determine where fuel-client is installed, if
>>>> it's master
>>>> node, we can perform installation, if it isn't master node, we can show
>>>> user the
>>>> message, that in the current version remote plugin installation is not
>>>> supported.
>>>> In the next versions if we implement plugin manager (which is separate
>>>> service
>>>> for plugins management) we will be able to do it remotely.
>>>>
>>>> >> How are we planning to distribute fuel plugin builder and its
>>>> >> updates?
>>>>
>>>> Yes, as Mike mentioned our plan is to release it on PyPi which is python
>>>> packages
>>>> repository, so any developer will be able to run `pip install fpb` and
>>>> get the tool.
>>>>
>>>> >> What happens if an error occurs during plugin installation?
>>>>
>>>> Plugins installation process is very simple, our plan is to have some
>>>> kind of transaction,
>>>> to make it atomic.
>>>>
>>>> 1. register plugin via API
>>>> 2. copy the files
>>>>
>>>> In case of error on the 1st step, we can do nothing, in case of error on
>>>> the 2nd step,
>>>> remove files if there are any, and delete a plugin via rest api. And
>>>> show user a message.
>>>>
>>>> >> What happens if an error occurs during plugin execution?
>>>>
>>>> In the first iteration we are going to interrupt deployment if there are
>>>> any errors for plugin's
>>>> tasks, also we are thinking how to improve it, for example we wanted to
>>>> provide a special
>>>> flag for each task, like fail_deploument_on_error, and only if it's
>>>> true, we fail deployment in
>>>> case of failed task. But it can be tricky to implement, it requires to
>>>> change the current
>>>> orchestrator/nailgun error handling logic. So, I'm not sure if we can
>>>> implement this logic in
>>>> the first release.
>>>>
>>>> Regarding to meaningful error messages, yes, we want to show the user,
>>>> which plugin
>>>> causes the error.
>>>>
>>>> >> Shall we consider a separate place in UI (tab) for plugins?
>>>>
>>>> +1 to Mike's answer
>>>>
>>>> >> When are we planning to focus on the 2 plugins which were identified
>>>> >> as must-haves
>>>> >> for 6.0? Cinder & LBaaS
>>>>
>>>> For Cinder we are going to implement plugin which configures GlusterFS
>>>> as cinder backend,
>>>> so, if user has installed GlusterFS cluster, we can configure our cinder
>>>> to work with it,
>>>> I want to mention that we don't install GlusterFS nodes, we just
>>>> configure cinder to work
>>>> with user's GlusterFS cluster.
>>>> Stanislaw B. already did some scripts which configures cinder to work
>>>> with GlusterFS, so
>>>> we are on testing stage.
>>>>
>>>> Regarding to LBaaS, Stanislaw B. did multinode implementation, ha
>>>> implementation is tricky
>>>> and requires some additional work, we are working on it.
>>>>
>>>> Nathan's questions:
>>>>
>>>> Looks like Mike answered UI related questions.
>>>>
>>>> >> Do we offer any kind of validation for settings on plug-ins?   Or
>>>> >> some way to for the developer
>>>> >> to ensure that setting that cannot be default or computed get
>>>> >> requested for the plug-in?
>>>>
>>>> Yes, each field can have regexp which is used during the validation.
>>>>
>>>> Mike's questions:
>>>>
>>>> >> One minor thing from me, which I forgot to mention during the demo:
>>>> >> verbosity of fpb run. I
>>>> >> understand it might sound like a bikeshedding now, but I believe if
>>>> >> we develop it right from
>>>> >> the very beginning, then we can save some time later. So I would
>>>> >> suggest normal, short INFO
>>>> >> output, and verbose one with --debug.
>>>>
>>>> Agree.
>>>>
>>>> Thanks for your feedback,
>>>>
>>>> On Sun, Oct 19, 2014 at 1:11 PM, Mike Scherbakov
>>>> <mscherbakov at mirantis.com> wrote:
>>>>>
>>>>> Hi all,
>>>>> I moved this conversation to openstack-dev to get a broader audience,
>>>>> since we started to discuss technical details.
>>>>>
>>>>> Raw notes from demo session:
>>>>> https://etherpad.openstack.org/p/cinder-neutron-plugins-second-demo.
>>>>>
>>>>> Let me start answering on a few questions below from Roman & Nathan.
>>>>>>
>>>>>> How are we planning to distribute fuel plugin builder and its updates?
>>>>>> Ideally, it should be available externally (outside of master node). I don't
>>>>>> want us to repeat the same mistake as we did with Fuel client, which doesn't
>>>>>> seem to be usable as an external dependency.
>>>>>
>>>>> The plan was to have Fuel Plugin Builder (fpb) on PyPI. Ideally it
>>>>> should be backward compatible with older Fuel release, i.e. when there is
>>>>> Fuel 7.0 out, you should be still able to create plugin for Fuel 6.0. If
>>>>> that it is going to be overcomplicated - I suggested to produce fpb for
>>>>> every Fuel release, and name it like fpb60, fpb61, fpb70, etc. Then it
>>>>> becomes easier to support and maintain plugin builders for certain versions
>>>>> of Fuel.
>>>>> Speaking about Fuel Client - there is no mistake. It's been discussed
>>>>> dozens of times, it's just lack of resources to make it on PyPI as well as
>>>>> to fix a few other things. I hope it could be done as part of efforts from
>>>>> [2].
>>>>>
>>>>>> - Perhaps we have a separate settings tab just for Plug-Ins?    For
>>>>>> some complex plug-ins, they might require a dedicated tab.   If we have too
>>>>>> many tabs it could get messy.
>>>>>
>>>>>
>>>>>>
>>>>>> Shall we consider a separate place in UI (tab) for plugins? Settings
>>>>>> tab seems to be overloaded.
>>>>>
>>>>>
>>>>> This is certainly under planning and discussion for future releases.
>>>>> See [1], for example. For 6.0, we agreed that we can just extend existing
>>>>> Settings tab with plugins-related fields.
>>>>>
>>>>> One minor thing from me, which I forgot to mention during the demo:
>>>>> verbosity of fpb run. I understand it might sound like a bikeshedding now,
>>>>> but I believe if we develop it right from the very beginning, then we can
>>>>> save some time later. So I would suggest normal, short INFO output, and
>>>>> verbose one with --debug.
>>>>>
>>>>> Thanks for feedback folks!!!
>>>>>
>>>>> [1]
>>>>> https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg37196.html
>>>>> [2]
>>>>> https://www.mail-archive.com/openstack-dev@lists.openstack.org/msg37001.html
>>>>>
>>>>>
>>>>> ---------- Forwarded message ----------
>>>>> From: Nathan Trueblood <ntrueblood at mirantis.com>
>>>>> Date: Sat, Oct 18, 2014 at 3:24 AM
>>>>> Subject: Re: plugins
>>>>>
>>>>> Agreed - I thought this initial PoC was great.
>>>>>
>>>>> A few initial thoughts about settings in the UI and plug-in in general:
>>>>>
>>>>> - Perhaps we have a separate settings tab just for Plug-Ins?    For
>>>>> some complex plug-ins, they might require a dedicated tab.   If we have too
>>>>> many tabs it could get messy.
>>>>> - It seems like we should consider how we handle the VMWare settings in
>>>>> light of plug-ins as well.   Since with VMWare we have a lot of setting to
>>>>> configure and settings validation.
>>>>> - Do we offer any kind of validation for settings on plug-ins?   Or
>>>>> some way to for the developer to ensure that setting that cannot be default
>>>>> or computed get requested for the plug-in?
>>>>>
>>>>> - We need to think carefully about both the plug-in developer
>>>>> experience (how hard to test, get error messages, etc) and the experience
>>>>> for the user who deploys the plug-in into an environment.
>>>>>
>>>>>
>>>>> -Nathan
>>>>>
>>>>> On Fri, Oct 17, 2014 at 4:13 PM, Roman Alekseenkov
>>>>> <ralekseenkov at mirantis.com> wrote:
>>>>>>
>>>>>>
>>>>>> I watched both videos (creating a file with the text from UI &&
>>>>>> installing and starting a service).
>>>>>>
>>>>>> It looks pretty good!! Some initial feedback/questions:
>>>>>>
>>>>>> I like the fact that fuel plugin builder appends version to the name
>>>>>> and makes it "fuel-awesome-plugin-1.2.3.tar". The approach is similar to
>>>>>> Java/Maven and is a good one.
>>>>>> I feel like we should not require user to unpack the plugin before
>>>>>> installing it. Moreover, we may chose to distribute plugins in our own
>>>>>> format, which we may potentially change later. E.g. "lbaas-v2.0.fp". I'd
>>>>>> rather stick with two actions:
>>>>>>
>>>>>> Assembly (externally): fpb --build <name>
>>>>>> Installation (on master node): fuel --install-plugin <name>
>>>>>>
>>>>>> How are we planning to distribute fuel plugin builder and its updates?
>>>>>> Ideally, it should be available externally (outside of master node). I don't
>>>>>> want us to repeat the same mistake as we did with Fuel client, which doesn't
>>>>>> seem to be usable as an external dependency.
>>>>>> How do we handle errors?
>>>>>>
>>>>>> What happens if an error occurs during plugin installation?
>>>>>> What happens if an error occurs during plugin execution? Does it
>>>>>> (should it?) fail the deployment? Will we show user an error message with
>>>>>> the name of plugin that failed?
>>>>>>
>>>>>> Shall we consider a separate place in UI (tab) for plugins? Settings
>>>>>> tab seems to be overloaded.
>>>>>> When are we planning to focus on the 2 plugins which were identified
>>>>>> as must-haves for 6.0? Cinder & LBaaS
>>>>>>
>>>>>> Once again, great job guys!
>>>>>>
>>>>>> Thanks,
>>>>>> Roman
>>>>>>
>>>>>> On Fri, Oct 17, 2014 at 9:32 AM, Mike Scherbakov
>>>>>> <mscherbakov at mirantis.com> wrote:
>>>>>>>
>>>>>>> Thanks, Evgeny, excellent work!
>>>>>>> Roman, I believe we are "green" with the feature. Watch yourself.
>>>>>>>
>>>>>>> Mike Scherbakov
>>>>>>> #mihgen
>>>>>>>
>>>>>>> On Oct 17, 2014 8:25 PM, "Evgeniy L" <eli at mirantis.com> wrote:
>>>>>>>>
>>>>>>>> Hi guys, here are the videos from the demo
>>>>>>>>
>>>>>>>> Part 1
>>>>>>>> https://drive.google.com/file/d/0B7I3b5vI7ZYXUGY1QVYyX3NLTWc/view
>>>>>>>>
>>>>>>>> Part 2
>>>>>>>> https://drive.google.com/file/d/0B7I3b5vI7ZYXWkRmV05fT1VEQkk/view
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Mike Scherbakov
>>>>> #mihgen
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> --
>>> Vitaly Kramskikh,
>>> Software Engineer,
>>> Mirantis, Inc.
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Mike Scherbakov
>> #mihgen
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Dmitry Borodaenko