Open Stack

On Tue, Nov 25, 2014 at 1:39 AM, Evgeniy L <eli at mirantis.com> wrote:
>
>
> On Tue, Nov 25, 2014 at 10:40 AM, Andrew Woodward <xarses at gmail.com> wrote:
>>
>> On Mon, Nov 24, 2014 at 4:40 AM, Evgeniy L <eli at mirantis.com> wrote:
>> > Hi Andrew,
>> >
>> > Comments inline.
>> > Also could you please provide a link on OpenStack upgrade feature?
>> > It's not clear why do you need it as a plugin and how you are going
>> > to deliver this feature.
>> >
>> > On Sat, Nov 22, 2014 at 4:23 AM, Andrew Woodward <xarses at gmail.com>
>> > wrote:
>> >>
>> >> So as part of the pumphouse integration, I've started poking around
>> >> the Plugin Arch implementation as an attempt to plug it into the fuel
>> >> master.
>> >>
>> >> This would require that the plugin install a container, and some
>> >> scripts into the master node.
>> >>
>> >> First look:
>> >> I've looked over the fuel plugins spec [1] and see that the install
>> >> script was removed from rev 15 ->16 (line 134) This creates problems
>> >> do to the need of installing the container, and scripts so I've
>> >> created a bug [2] for this so that we can allow for an install script
>> >> to be executed prior to HCF for 6.0.
>> >
>> >
>> > Yes, it was removed, but nothing stops you from creating the install
>> > script and putting it in tarball, you don't need any changes in the
>> > current implementation.
>>
>> how would it be executed? the plugin loading done by fuel-client
>> doesn't cover this.
>
>
> Manually untar and run your script, as it was designed before we implemented
> more user friendly approach.

Needs some TLC, but here is a working patch

https://review.openstack.org/137301
>
>>
>> >
>> > The reasons why it was done this way, see in separate mailing thread
>> > [1].
>> >
>> > [1]
>> >
>> > http://lists.openstack.org/pipermail/openstack-dev/2014-October/049073.html
>> >
>> >>
>> >>
>> >> Looking into the implementation of the install routine [3] to
>> >> implement [2], I see that the fuelclient is extracting the tar blindly
>> >> (more on that at #3) on the executor system that fuelclient is being
>> >> executed from. Problems with this include 1) the fuelclient may not
>> >> root be privileged (like in Mirantis OpenStack Express) 2) the
>> >> fuelclient may not be running on the same system as nailgun 3) we are
>> >> just calling .extractall on the tarball, this means that we haven't
>> >> done any validation on the files coming out of the tarball. We need to
>> >> validate that 3.a) the tarball was actually encoded with the right
>> >> base path 3.b) that the tasks.yaml file is validated and all the noted
>> >> scripts are found. Really, the install of the plugin should be handled
>> >> by the nailgun side to help with 1,2.
>> >
>> >
>> > 1. if you have custom installation you have to provide custom
>> > permissions
>> >     for /var/www/nailgun/plugins directory
>> > 2. you are absolutely right, see the thread from above why we decided to
>> > add
>> >     this feature even if it was a wrong decision from architecture point
>> > of
>> > view
>> > 3. "haven't done any validation" - not exactly, validation is done on
>> > plugin
>> >     building stage, also we have simple validation on plugin
>> > installation
>> > stage on
>> >     Nailgun side (that data are consistent from nailgun point of view).
>> > There are
>> >     several reasons why it was done mainly on fuel-plugin-builder side:
>> >       a. plugin is validated before it's installed (it dramatically
>> > simplifies development)
>> >       b. also you can check that plugin is valid without plugin
>> > building,
>> >           use 'fpb --check fuel_plugin_name' parameter
>> >       c. faster fixes delivery, if there is a bug in validation (we had
>> > several of them
>> >           during the development in fuel-plugin-builder), we cannot just
>> > release new
>> >           version of fuel, but we can do it with fuel-plugin-builder, we
>> > had

I've already found some validation bugs in fpb

https://bugs.launchpad.net/fuel/+bug/1396491
https://bugs.launchpad.net/fuel/+bug/1396495 (
https://review.openstack.org/137304 )
https://bugs.launchpad.net/fuel/+bug/1396499

>> > 2 releases [1].
>> >           For more complicated structures you will have bugs in
>> > validation
>> > for sure.
>> >       d. if we decide to support validations on both sides, we will come
>> > up
>> > with a lot of bugs
>> >           which are related to desynchronization of validators between
>> > Nailgun and fuel-plugin-builder
>>
>> the main validation points that should be done by nailgun is to verify
>> that the paths are correct. i.e.
>> * the tar ./<folder> == metadata.yaml['name']
>> * tasks.yaml + metadata.yaml refer to valid paths for "cmd",
>> "deployment_scripts_path", "repository_path"
>>
>> Rright now there is no contract between the user building the plugin
>> with fpb, vs adding all the files to a tarball. if fpb is supposed to
>> be doing this, then there should be some form of signature that can be
>> parsed to ensure that these items have been pre-validated and the
>> package wasn't modified, or built by hand. Something that would be
>> easy, and cheap would be something like 'cat metdata.yaml tasks.yaml |
>> md5sum >md5sum' and validate this when we load the package. It also
>> gives us a starting point for other signers.
>
>
> Do we really want to cover cases when somebody made a tar-ball manually
> and gave it to a user without any testing?
> I don't think so.

The code I've dug into shows that we are not validating file paths, so
yes; something needs to test that the paths are valid. And if nailgun
isn't validating things like this then we need to implement signing so
we know it wasn't modified, or had lost bits.

>
>>
>> Alternatly, we would use fpb to validate the package prior to
>> installing it into nailgun.
>
>
> I'm not sure if it's a good option, fpb should not be included in the iso,
> it has separate release cycle, we shouldn't increase the amount
> of packages which we update during the upgrade, because of
> rollback problems with packages which are not in the containers.
>
>>
>>
>> >
>> > [1]
>> >
>> > https://github.com/stackforge/fuel-plugins/blob/master/fuel_plugin_builder/CHANGELOG.md
>> >
>> >>
>> >>
>> >> Whats next?
>> >> There are many parts of PA that need to be extended, I think that
>> >> these are the ones that we must tackle next to cover the most cases
>> >> a) plugin packaging: it appears that non of the "core plugins" (those
>> >> in fuel-plugins) are bundled into the iso.
>> >> b) plugin signing: we cant have "core plugins" with out some method of
>> >> testing, certifying, and signing them so that we can know that they
>> >> are trusted.
>> >>
>> >> with the help of granular roles:
>> >> c) the ability to replace or add new granular roles
>> >> d) the ability to add or modify real roles
>> >>
>> >> with the help of advanced networks:
>> >> e) add new network roles
>> >>
>> >> At some point soon, we also need to discuss making it easier to find a
>> >> catalog of modules and pull them from it, but this is less important
>> >> than the above
>> >>
>> >> [1]
>> >>
>> >> https://review.openstack.org/#/c/125608/15..16/specs/6.0/cinder-neutron-plugins-in-fuel.rst
>> >> [2] https://bugs.launchpad.net/fuel/+bug/1395228
>> >> [3]
>> >>
>> >> https://github.com/stackforge/fuel-web/blob/master/fuelclient/fuelclient/objects/plugins.py#L49
>> >>
>> >> --
>> >> Andrew
>> >> Mirantis
>> >> Ceph community
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > OpenStack-dev mailing list
>> > OpenStack-dev at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >
>>
>>
>>
>> --
>> Andrew
>> Mirantis
>> Ceph community
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Andrew
Mirantis
Ceph community